Steam Scraper Proof of concept for mgalaxy

Hobbes · Thu Oct 07, 2021 4:58 pm

Sooooo.....
A few months back I asked for a steam scraper for steam games. I see the mgalaxy administrator is hard at work (version 8.5) so since I am laid up with nasty vertigo and I needed to do something besides...you know... work and was interested in sharpening my Python skills and starting the leg work for the mgalaxy team I came up with this sub alpha version of a steam game scraper. Results below.

-- WARNING: THIS IS SUB ALPHA CODE, DO NOT USE IF YOU DON'T KNOW PYTHON - HELL EVEN IF YOU DO (I have vertigo and am on a LOT of drugs) ... IT IS PURELY A PROOF OF CONCEPT --

-What it does-
* Creates shortcuts of your steam library
* Downloads images for you steam library (excluding screen shots)
* Puts images in appropriate Meta directory

-What it doesn't-
* Create screenshots
* Update games database with correct descriptions

-Issues-
* This works by running steam silently in the background ( some of my titles had issues just running exe ) and then kicking off by appid. This means that currently mgalaxy will not close the app correctly and you will have to open a minimized mgalaxy after closing. (Help! Any ideas? )
* Uses a lot of files, two external program (steamgrid_windows and steam.exe),and is generally clunky

-Lessons learned (useful for mgalaxy creators)-
* a workable version in mgalaxy_runway is totally reasonable without the extra headaches, it would need to use the following:
a. the public games list xml available for a user in steam: https://steamcommunity.com/id/<profileID>/games?xml=1
- this would pull the game name and app id
b. use of the steamgriddb api: https://www.steamgriddb.com/api/v2
- this would easily pull appropriate graphics
c. use of steamapi : https://steamapis.com/docs/market#app
- to get the description and the screenshots (possibly all graphics depending on how you feel about image choices)

Almost all of this just uses web apis (except a.) and are less load limited than the steamworks apis

* Next time I will use the web apis as opposed to external apps and crappy hooks
* BIG ISSUE - it runs apps pretty quick but has an issue while closing (see under issues) can this be fixed with a script?
* Is there a file location one can put game information?

Aeliss · Tue Oct 12, 2021 4:04 pm

* This works by running steam silently in the background ( some of my titles had issues just running exe ) and then kicking off by appid. This means that currently mgalaxy will not close the app correctly and you will have to open a minimized mgalaxy after closing. (Help! Any ideas? )

Can you explain more the issue ?

On some EXE game, I have this kind of issue, So I have used autoit script to run bat file, so it was possible to monitor cmd.exe instead of the EXE. Using autoit script is nice, because mGalaxy will monitor the script, and you can do all you want in it, and close it only if you want.

If it can help you

Code: Select all

;Parsing
Local $count = StringInStr($CmdLineRaw, ".exe", 0 , 1, 1)
local $app = StringLeft ( $CmdLineRaw, $count + 4 )
local $command = StringTrimLeft( $CmdLineRaw, $count + 4 )
if not $command then exit 0

;Get more info
$count = StringInStr($command, '"', 0 , 1, 1)
local $link = StringTrimLeft ( $command, $count  )
$link = StringTrimRight ( $link, 1 )
Local $aDetails = FileGetShortcut($link)
If @error Then exit
local $tmp = StringSplit($aDetails[0], "")
local $apptorun = $tmp[$tmp[0]]


;run command
;Run ('"' & $app & '" ' & $command)
ShellExecute('"' & $link & '"')

;wait for application launched with protection loop
;wait for 5 s and exit if no application found
local $protloop = 10
While not(ProcessExists($apptorun))
    $protloop = $protloop - 1
   if ($protloop < 0) then exit 0
   sleep(500)
WEnd

;try to put in front
local $PID = GetProcessPID($apptorun)
local $WinName =  ProcessGetWindow($PID)
WinSetState($WinName,"",@SW_SHOW)
WinSetState($WinName,"",@SW_ENABLE)

;wait for close
While (ProcessExists($apptorun) or (ProcessExists("cmd.exe")))
   sleep(500)
WEnd

;To resolve bug in taito  Xom Tetris gran master
local $list = ProcessList("game.exe")
if $list[0][0] > 0 Then
   local $PID = $list[1][1]
   ProcessWaitClose($PID)
endif

Exit 0




;*********************************************************
;functions
;*********************************************************

Func GetProcessPID($name)
   Local $list = ProcessList($name)
   For $i = 1 To $list[0][0]
       If $list[$i][0] <> "" then
          Return $list[$i][1]
        Endif
   Next
   return 0
endFunc

Func ProcessGetWindow($PId)
    If IsNumber($PId) = 0 then
        SetError(1)
    Else
        Local $WinList = WinList()

        For $i = 1 To $WinList[0][0]
            If WinGetProcess($WinList[$i][0], "") = $PId Then
                return $WinList[$i][0]
            EndIf
        Next
        Return 0
    EndIf
 EndFunc
 
 
 
;~   
;~ Func _WinWaitActivePID($iPid)
;~    While 1
;~        Local $list = WinList()
;~        For $i = 1 To $list[0][0]
;~            If $list[$i][0] <> "" And IsVisible($list[$i][1]) Then
;~                If WinGetProcess($list[$i][1]) = $iPid AND WinActive($list[$i][1]) Then Return $list[$i][0]
;~            EndIf
;~        Next
;~    WEnd
;~ EndFunc
;~      
;~ Func IsVisible($handle)
;~     If BitAND(WinGetState($handle), 2) Then
;~         Return 1
;~     Else
;~         Return 0
;~     EndIf
;~  EndFunc

Hobbes · Wed Oct 13, 2021 5:24 pm

Ok questions
1. How would I implement the autoit script. I created a special system for "Steam". should it be in Steam [Script].au3?
2. I have an idea for the script but don't know the particulars... how would I figure out LastItem (Last Item being the pid of the last opened window)

Code: Select all

While ProcessExists ( %LastItem% )
   sleep(500)
WEnd
Exit 0

WinActivate("mGalaxy")

3. NOTE: I should have a python code without steamgriddb and with screenshots and a variable for description by the end of the week

Hobbes · Thu Oct 14, 2021 5:30 am

Ok finished version 2

-- WARNING: THIS IS SUB ALPHA CODE, DO NOT USE IF YOU DON'T KNOW PYTHON - HELL EVEN IF YOU DO (I have vertigo and am on a LOT of drugs) ... IT IS PURELY A PROOF OF CONCEPT --

-What it does-
* Creates shortcuts of your steam library
* Downloads images for you steam library (excluding screen shots)
* Puts images in appropriate Meta directory
* Downloads screenshots [NEW]
* Downloads movies [NEW]
-What it doesn't-
* Update games database with correct descriptions (although there is a variable for downloading descriptions in the script

-Issues-
* This works by running steam silently in the background ( some of my titles had issues just running exe ) and then kicking off by appid.
This means that currently mgalaxy will not close the app correctly and you will have to open a minimized mgalaxy after closing.
(Help! Any ideas? )

-Lessons learned (useful for mgalaxy creators)-
* a workable version in mgalaxy_runway is totally reasonable without the extra headaches, it would need to use the following:
a. the public games list xml available for a user in steam: https://steamcommunity.com/id/<profileID>/games?xml=1
- this would pull the game name and app id
b. use of the steamgriddb api: https://www.steamgriddb.com/api/v2
- this would easily pull appropriate graphics
c. pulling steam images, movies, and info from akami (no api key needed)
- to get the graphics, descriptions, screenshots, and movies

* ISSUE 1 - it runs apps pretty quick but has an issue while closing (-Aeliss- please check out the post below this one)
* ISSUE 2 - Is there a file location one can put game information?

Hobbes · Thu Oct 14, 2021 5:32 am

Image to show what everything looks like

Aeliss · Thu Oct 14, 2021 2:58 pm

How would I implement the autoit script. I created a special system for "Steam". should it be in Steam [Script].au3?

Yep the name seem good, on my side I m using the same structure for all systems, so I have a structure like that

Folder "Shortcut", inside a folder "lnk" and inside all *.lnk file (the "rom" folder), cmd.exe file as executable (the emulator executable) and the script file compiled inside this folder (but it s because was the old working mode on mGalaxy)

You can use direclty the script as executable.

I have an idea for the script but don't know the particulars... how would I figure out LastItem (Last Item being the pid of the last opened window)

You can't use the fonction GetProcessPID ? this one make a search according to a name and return a PID. You can't use steam executable (it stay open), but the game's name is not working ?

Hobbes · Sat Oct 16, 2021 11:40 pm

Ok finished version 3

-- WARNING: THIS IS SUB ALPHA CODE, DO NOT USE IF YOU DON'T KNOW PYTHON - HELL EVEN IF YOU DO (I have vertigo and am on a LOT of drugs) ... IT IS PURELY A PROOF OF CONCEPT --

-What it does-
* Creates shortcuts of your steam library
* Downloads images for you steam library (excluding screen shots)
* Puts images in appropriate Meta directory
* Downloads screenshots
* Downloads movies
* Updates database with creation date, and overiew [NEW]
* Runs in command line as an exe file [NEW]
* Uses a config.ini file for all file locations [NEW[
* Pulls game list xml form your public steam profile (see instructions) [NEW]

-Issues-
* This works by running steam silently in the background ( some of my titles had issues just running exe ) and then kicking off by appid.
This means that currently you will have to open a minimized mgalaxy after closing a game.
(Help! Any ideas? )

-Lessons learned (useful for mgalaxy creators)-
* a wo
rkable version in mgalaxy_runway is totally reasonable without the extra headaches, it would need to use the following:
a. the public games list xml available for a user in steam: https://steamcommunity.com/id/<profileID>/games?xml=1
- this would pull the game name and app id
b. use of the steamgriddb api: https://www.steamgriddb.com/api/v2
- this would easily pull appropriate graphics
c. pulling steam images, movies, and info from akami (no api key needed)
- to get the graphics, descriptions, screenshots, and movies

* ISSUE 1 - it runs apps pretty quick but has an issue while closing - see above, any help would be a godsend
* ISSUE 2 - created a .exe (yay pyinstaller) but it is slighty too big to add as an attachment ( moderators what
would you like me to do with it? Also does any of this help you guys with supporting steam? I'm pretty much
done except for more robust error messages and fixing the closing app to minmized mgalaxy issue (Issue 1)