Pretty severe temporary file and memory usage problem making delta map

[Rotareneg]

When making a new delta map from the attached "sce_Platoon Attack" published delta map, SB's memory use will grow to enormous amounts while also creating an 22.8 GB "ScaniaSE.thm" file in the system temporary folder. After exiting the map editor and SB, that huge ScaniaSE.thm file was left behind in the temporary folder.

 

The attached ScaniaSE.7z file is that ScaniaSE.thm file, compressed down to only 4 MB. It's password protected to avoid potential memory problems if an automatic virus scanner tried to open it, password is:

 

ScaniaSE

 

Below is the log showing how the SB process memory and page file grew, including a memory error which probably happened because the C : drive (which otherwise has about 40 GB of free space) got filled up by that temporary file and the page file.

[21:16:43,538] CGame WARN : -------------------------------
[21:16:43,538] CGame WARN : Steel Beasts Log File v4.157
[21:16:43,538] CGame WARN : Thu Aug  8 21:16:43 201 / 5784
[21:16:43,538] CGame WARN : -------------------------------
[21:16:43,538] CGame WARN : CPU does not have AVX support
[21:16:43,538] CGame INFO : +--+ MEMORY USAGE INFO (SteelBeasts START):
[21:16:43,538] CGame INFO : |  | Process: 4.3 GB (0 % of 128.0 TB total, 128.0 TB free) 4.3 GB min/4.2 GB max
[21:16:43,538] CGame INFO : |  | Physical: 1.9 GB (24 % of 8.0 GB total, 6.1 GB free) 1.8 GB min/1.9 GB max
[21:16:43,538] CGame INFO : +--+ Pagefile: 2.9 GB (26 % of 11.2 GB total, 8.4 GB free) 2.8 GB min/2.9 GB max
[21:17:03,565] CGame INFO : +--+ MEMORY USAGE INFO (LoadFileInMapEditor start):
[21:17:03,565] CGame INFO : |  | Process: 6.0 GB (0 % of 128.0 TB total, 128.0 TB free) 4.3 GB min/6.1 GB max
[21:17:03,566] CGame INFO : |  | Physical: 3.3 GB (41 % of 8.0 GB total, 4.7 GB free) 1.8 GB min/3.4 GB max
[21:17:03,566] CGame INFO : +--+ Pagefile: 4.5 GB (40 % of 11.2 GB total, 6.8 GB free) 2.8 GB min/4.6 GB max
[21:17:39,998] CGame INFO : +--+ MEMORY USAGE INFO (LoadFileInMapEditor end):
[21:17:40,004] CGame INFO : |  | Process: 14.6 GB (0 % of 128.0 TB total, 128.0 TB free) 4.3 GB min/14.6 GB max
[21:17:40,004] CGame INFO : |  | Physical: 7.4 GB (93 % of 8.0 GB total, 575 MB free) 1.8 GB min/7.4 GB max
[21:17:40,004] CGame INFO : +--+ Pagefile: 11.0 GB (77 % of 14.3 GB total, 3.3 GB free) 2.8 GB min/11.0 GB max
[21:19:25,300] CGame WARN : TiledBitmap::get_block, error allocating memory. Will re-attempt after processing the cache.
[21:19:42,167] CGame INFO : +--+ MEMORY USAGE INFO (SteelBeasts END):
[21:19:42,167] CGame INFO : |  | Process: 22.7 GB (0 % of 128.0 TB total, 128.0 TB free) 4.3 GB min/23.8 GB max
[21:19:42,167] CGame INFO : |  | Physical: 5.6 GB (70 % of 8.0 GB total, 2.4 GB free) 1.8 GB min/7.5 GB max
[21:19:42,167] CGame INFO : +--+ Pagefile: 18.0 GB (82 % of 21.9 GB total, 3.9 GB free) 2.8 GB min/19.5 GB max
[21:19:44,822] CGame INFO : Done

I also noticed Windows recorded a resource exhaustion event in the event log:

 

Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: SBProPE64cm.exe (5784) consumed bytes, MsMpEng.exe (3264) consumed 187203584 bytes, and explorer.exe (9932) consumed 113659904 bytes.

sce_Platoon_Attack.7z ScaniaSE.7z

Edited August 9, 2019 by Rotareneg
Stupid forum converted C: to an emoticon. *sigh*

[Gibsonm]

Perhaps fixed in 4.159?

[Volcano]

Just curious, were you leveling roads or do any kind of height adjustments to the map before publishing them?

[Rotareneg]

The attached map does have height adjustments on streams and leveling around buildings, but when publishing that the first time it made a huge temporary file in the target folder, not the system temporary file folder. I didn't notice any problems then, probably because there was enough space on the C drive for the page file to expand as needed without the huge temporary file also residing there.

[Ssnake]

Case number: 7822.

[Bond_Villian]

@RotarenegWhere is this temp file located typically? Curious so i can check if im getting anything similar ( my computer becomes extrememy slow/unresponsive when map editor is running).

Cheers

[Rotareneg]

%localappdata%\Temp

 

It'll have all sorts of random stuff in it (which isn't normally an issue as Windows generally clears old unused files out of there automatically.) You'd be looking for very large multi-gigabyte files named after SB height maps. It may not be an issue if you haven't used any map editing functions that change the terrain height like raising or leveling roads or leveling buildings, I don't think those huge temporary files get created unless you've done that.

[Bond_Villian]

Ok taa

[Rotareneg]

Wow... I added an extra page file on another drive and set the temporary folder to be compressed which spared enough room on my C drive to properly load and save that map again. SB peaked at 34.5 GB of memory used during the process, with the page file usage at 32.6 GB. After saving it the temporary file was cleared out of the temp folder and the more expected ScaniaSE.dhnt file was placed in the new map folder. Good news is NTFS compression works extremely well with those .dhnt files, packing a 22.8 GB file down in 8.37 MB, and in the process greatly speeding up the access speed of the file compared to reading/writing the uncompressed file from HD.

Edited August 9, 2019 by Rotareneg

[Volcano]

17 hours ago, Rotareneg said:

The attached map does have height adjustments on streams and leveling around buildings, but when publishing that the first time it made a huge temporary file in the target folder, not the system temporary file folder. I didn't notice any problems then, probably because there was enough space on the C drive for the page file to expand as needed without the huge temporary file also residing there.

Yes, this is exactly why I asked.

 

As far as I am aware, this ballooing of size is normal because (layman's terms here - as I don't know the exact details myself) it has to unpack the full height data, which is very large, in order to make changes to the data.  Once you publish it, then it packs it all back down to the usual size again.  This was described to me as being the normal way it works. Still, we will ask the programmer that knows how it works and will see if it is an issue or not.

 

Obviously a bug would be if it doesn't actually get rid of the temporary file, though.  Then again, maybe I missed the point here.