Muldersoft/TimedExec
Muldersoft/TimedExec
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 2 Wiki Activity

Use GetProcessTimes() function for time measurements + added option to measure CPU time instead of wall-clock time.

Browse Source
This commit is contained in:
LoRd_MuldeR 2023-04-12 23:09:39 +02:00
parent 5464c19013
commit 0398acf4b1
11 changed files with 311 additions and 84 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

27
README.md
View File

@ -1,13 +1,12 @@
% ![](img/timedexec/banner.jpg)
TimedExec – README
% by LoRd_MuldeR &lt;<mulder2@gmx>&gt; | <http://muldersoft.com/>
![TimedExec](img/timedexec/banner.jpg)
by LoRd_MuldeR &lt;<mulder2@gmx>&gt; | <http://muldersoft.com/>
Introduction
============
**TimedExec** is a small utility for *benchmarking* command-line programs. It will *execute* the specified program with the specified command-line arguments and then *measure* the time that it takes for the execution to complete. In order to obtain *accurate* results, all measurements are implemented via *high-resolution* performance timers. And, since program execution times unavoidably are subject to certain variations (e.g. due to environmental noise), each test will be repeated *multiple* times. The number of metering passes can be configured as desired. Optionally, a number of "warm-up" passes can be performed *prior to* the first metering pass. The warm-up passes prevent caching effects from interfering with the execution times.
TimedExec will then compute the ***mean*** execution time as well as the ***median*** execution time of all metering passes. It will also record the *fastest* and *slowest* execution time that has been measured. Furthermore, TimedExec computes the *standard error* in order to determine ***confidence intervals*** from the benchmarking results^[[Konfidenzintervalle so einfach wie möglich erklärt](http://www.uni-siegen.de/phil/sozialwissenschaften/soziologie/mitarbeiter/ludwig-mayerhofer/statistik/statistik_downloads/konfidenzintervalle.pdf)]. These are the *ranges* which contain the program's “real” average execution time (expected value), *with very high probability*. All results will be saved to a log file.
TimedExec will then compute the ***mean*** execution time as well as the ***median*** execution time of all metering passes. It will also record the *fastest* and *slowest* execution time that has been measured. Furthermore, TimedExec computes the *standard error* in order to determine [***confidence intervals***](http://www.uni-siegen.de/phil/sozialwissenschaften/soziologie/mitarbeiter/ludwig-mayerhofer/statistik/statistik_downloads/konfidenzintervalle.pdf) from the benchmarking results. These are the *ranges* which contain the program's “real” average execution time (expected value), *with very high probability*. All results will be saved to a log file.
Usage Instructions
@ -19,8 +18,8 @@ Usage Instructions
```
===============================================================================
Timed Exec - Benchmarking Utility, Version 1.03
Copyright (c) 2018 LoRd_MuldeR <mulder2@gmx.de>. Some rights reserved.
Timed Exec - Benchmarking Utility, Version 1.05
Copyright (c) 2023 LoRd_MuldeR <mulder2@gmx.de>. Some rights reserved.
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License <http://www.gnu.org/>.
@ -35,8 +34,19 @@ Influential environment variables:
TIMED_EXEC_WARMUP_PASSES - Number of warm-up passes (default: 1)
TIMED_EXEC_LOGFILE - Log-File Name (default: "TimedExec.log")
TIMED_EXEC_NO_CHECKS - Set this to *disable* exit code checks
TIMED_EXEC_CLOCK_TYPE - The type of clock used for measurements
```
Clock Types
-----------
*TimedExec* supports the following clock types:
- `WALLCLOCK` &ndash; the wall-clock time, also known as *elapsed real time* (default)
- `CPU_TOTAL` &ndash; total CPU time, i.e. sum of CPU time spent in "user" *and* "kernel" modes
- `CPU_USER` &ndash; CPU time spent in "user" mode only
- `CPU_KERNEL` &ndash; CPU time spent in "kernel" mode only
Usage Example
-------------
@ -64,13 +74,14 @@ Median Execution Time : 20.522 seconds
Standard Deviation : 7.697 seconds
Standard Error : 3.848 seconds
Fastest / Slowest Pass : 19.739 / 37.916 seconds
Active Clock Type : WALLCLOCK (0)
===============================================================================
```
Interpretation
--------------
When comparing measurement results, the ***mean*** (average) execution time may seem like the most obvious choice. However, it has to be noted that the *mean* of a data sample is highly sensitive to “outliers” and therefore can be misleading! This is especially true, when there exists a lot of variation in the data sample. Consequently, comparing the ***median*** execution times usually is the preferable choice. That is because the *median* of a data sample is much more robust against outliers.
When comparing measurement results, the ***mean*** (average) execution time may seem like the most obvious choice. However, it has to be noted that the *mean* of a data sample is highly sensitive to “outliers” and therefore can be misleading! This is especially true, when there exists a lot of variation in the data sample. Consequently, comparing the ***median*** execution times often is the better choice. That is because the *median* of a data sample is much more robust against outliers.
Furthermore, it is important to keep in mind that the *mean* (or *median*) execution time computed from a limited number of metering passes only yields an ***estimate*** of the program's “real” average execution time (expected value). The “real” value can only be determined accurately from an *infitinte* number of metering passes &ndash; which is **not** possible in practice. In this situation, we can have a look at the ***confidence intervals***. These intervals contain the “real” value, *with very high probability*. The most commonly used *confidence interval* is the “95%” one (higher confidence means broader interval, and vice versa).
@ -94,7 +105,7 @@ TimedExec is released under the terms of the [GNU General Public License](http:/
```
Timed Exec - Command-Line Benchmarking Utility
Copyright (c) 2018 LoRd_MuldeR <mulder2@gmx.de>. Some rights reserved.
Copyright (c) 2018-2023 LoRd_MuldeR <mulder2@gmx.de>. Some rights reserved.
This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License

9
TimedExec_VS2010.vcxproj
View File

@ -21,6 +21,15 @@
<ItemGroup>
<ClCompile Include="src\TimedExec.cpp" />
</ItemGroup>
<ItemGroup>
<ResourceCompile Include="res\version.rc" />
</ItemGroup>
<ItemGroup>
<Manifest Include="res\compatibility.manifest" />
</ItemGroup>
<ItemGroup>
<ClInclude Include="src\Version.h" />
</ItemGroup>
<PropertyGroup Label="Globals">
<ProjectGuid>{7BB6FBD8-4531-4CBB-B9EE-D440ACD7BCEC}</ProjectGuid>
<Keyword>Win32Proj</Keyword>

15
TimedExec_VS2010.vcxproj.filters
View File

@ -19,4 +19,19 @@
<Filter>Source Files</Filter>
</ClCompile>
</ItemGroup>
<ItemGroup>
<ResourceCompile Include="res\version.rc">
<Filter>Resource Files</Filter>
</ResourceCompile>
</ItemGroup>
<ItemGroup>
<Manifest Include="res\compatibility.manifest">
<Filter>Resource Files</Filter>
</Manifest>
</ItemGroup>
<ItemGroup>
<ClInclude Include="src\Version.h">
<Filter>Header Files</Filter>
</ClInclude>
</ItemGroup>
</Project>

20
TimedExec_VS2017.vcxproj
View File

@ -21,38 +21,48 @@
<ItemGroup>
<ClCompile Include="src\TimedExec.cpp" />
</ItemGroup>
<ItemGroup>
<ClInclude Include="src\Version.h" />
</ItemGroup>
<ItemGroup>
<ResourceCompile Include="res\version.rc" />
</ItemGroup>
<ItemGroup>
<Manifest Include="res\compatibility.manifest" />
</ItemGroup>
<PropertyGroup Label="Globals">
<ProjectGuid>{7BB6FBD8-4531-4CBB-B9EE-D440ACD7BCEC}</ProjectGuid>
<Keyword>Win32Proj</Keyword>
<RootNamespace>TimedExec</RootNamespace>
<ProjectName>TimedExec</ProjectName>
<WindowsTargetPlatformVersion>10.0.19041.0</WindowsTargetPlatformVersion>
</PropertyGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>true</UseDebugLibraries>
<CharacterSet>Unicode</CharacterSet>
<PlatformToolset>v141_xp</PlatformToolset>
<PlatformToolset>v141</PlatformToolset>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>true</UseDebugLibraries>
<CharacterSet>Unicode</CharacterSet>
<PlatformToolset>v141_xp</PlatformToolset>
<PlatformToolset>v141</PlatformToolset>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>false</UseDebugLibraries>
<WholeProgramOptimization>true</WholeProgramOptimization>
<CharacterSet>Unicode</CharacterSet>
<PlatformToolset>v141_xp</PlatformToolset>
<PlatformToolset>v141</PlatformToolset>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>false</UseDebugLibraries>
<WholeProgramOptimization>true</WholeProgramOptimization>
<CharacterSet>Unicode</CharacterSet>
<PlatformToolset>v141_xp</PlatformToolset>
<PlatformToolset>v141</PlatformToolset>
</PropertyGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
<ImportGroup Label="ExtensionSettings">
@ -151,6 +161,7 @@
<LinkTimeCodeGeneration>UseLinkTimeCodeGeneration</LinkTimeCodeGeneration>
<AdditionalDependencies>$(SolutionDir)\etc\EncodePointer\lib\EncodePointer.lib;Shell32.lib;Shlwapi.lib;%(AdditionalDependencies)</AdditionalDependencies>
<ImageHasSafeExceptionHandlers>false</ImageHasSafeExceptionHandlers>
<MinimumRequiredVersion>5.1</MinimumRequiredVersion>
</Link>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
@ -181,6 +192,7 @@
<LinkTimeCodeGeneration>UseLinkTimeCodeGeneration</LinkTimeCodeGeneration>
<ImageHasSafeExceptionHandlers>false</ImageHasSafeExceptionHandlers>
<AdditionalDependencies>Shell32.lib;;Shlwapi.lib;%(AdditionalDependencies)</AdditionalDependencies>
<MinimumRequiredVersion>5.2</MinimumRequiredVersion>
</Link>
</ItemDefinitionGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />

15
TimedExec_VS2017.vcxproj.filters
View File

@ -19,4 +19,19 @@
<Filter>Source Files</Filter>
</ClCompile>
</ItemGroup>
<ItemGroup>
<ClInclude Include="src\Version.h">
<Filter>Header Files</Filter>
</ClInclude>
</ItemGroup>
<ItemGroup>
<ResourceCompile Include="res\version.rc">
<Filter>Resource Files</Filter>
</ResourceCompile>
</ItemGroup>
<ItemGroup>
<Manifest Include="res\compatibility.manifest">
<Filter>Resource Files</Filter>
</Manifest>
</ItemGroup>
</Project>

BIN
res/appicon.ico Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 7.7 KiB

17
res/compatibility.manifest Normal file
View File

@ -0,0 +1,17 @@
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
<compatibility xmlns="urn:schemas-microsoft-com:compatibility.v1">
<application>
<!-- Windows Vista -->
<supportedOS Id="{e2011457-1546-43c5-a5fe-008deee3d3f0}"/>
<!-- Windows 7 -->
<supportedOS Id="{35138b9a-5d96-4fbd-8e2d-a2440225f93a}"/>
<!-- Windows 8 -->
<supportedOS Id="{4a2f28e3-53b9-4441-ba9c-d69d4a4a6e38}"/>
<!-- Windows 8.1 -->
<supportedOS Id="{1f676c76-80e1-4239-95bb-83d0f6d0da78}"/>
<!-- Windows 10 -->
<supportedOS Id="{8e0f7a12-bfb3-4fe8-b9a5-48fd50a15a9a}"/>
</application>
</compatibility>
</assembly>

68
res/version.rc Normal file
View File

@ -0,0 +1,68 @@
/////////////////////////////////////////////////////////////////////////////
//
// Microsoft Visual C++ generated resource script.
//
#define APSTUDIO_READONLY_SYMBOLS
#include "WinResrc.h"
#undef APSTUDIO_READONLY_SYMBOLS
#include "../src/Version.h"
#define ___FULL_VERSION_STR___(X) #X
#define __FULL_VERSION_STR__(X,Y,Z) ___FULL_VERSION_STR___(X.Y##Z)
#define _FULL_VERSION_STR_(X,Y,Z) __FULL_VERSION_STR__(X,Y,Z)
#define FULL_VERSION_STR _FULL_VERSION_STR_(VERSION_MAJOR,VERSION_MINOR_HI,VERSION_MINOR_LO)
/////////////////////////////////////////////////////////////////////////////
//
// Neutral resources
//
#ifdef _WIN32
LANGUAGE LANG_NEUTRAL, SUBLANG_NEUTRAL
#pragma code_page(1252)
#endif //_WIN32
/////////////////////////////////////////////////////////////////////////////
//
// Icon
//
101 ICON "appicon.ico"
/////////////////////////////////////////////////////////////////////////////
//
// Version
//
VS_VERSION_INFO VERSIONINFO
FILEVERSION VERSION_MAJOR, VERSION_MINOR_HI, VERSION_MINOR_LO, 0
PRODUCTVERSION VERSION_MAJOR, VERSION_MINOR_HI, VERSION_MINOR_LO, 0
FILEFLAGSMASK 0x17L
#ifdef _DEBUG
FILEFLAGS 0x3L
#else
FILEFLAGS 0x2L
#endif
FILEOS 0x40004L
FILETYPE 0x1L
FILESUBTYPE 0x0L
BEGIN
BLOCK "StringFileInfo"
BEGIN
BLOCK "000004b0"
BEGIN
VALUE "ProductName", "TimedExec"
VALUE "FileDescription", "TimedExec"
VALUE "ProductVersion", FULL_VERSION_STR
VALUE "FileVersion", FULL_VERSION_STR
VALUE "InternalName", "TimedExec"
VALUE "OriginalFilename", "TimedExec.exe"
VALUE "LegalCopyright", "Created by LoRd_MuldeR <MuldeR2@GMX.de>"
VALUE "CompanyName", "Muldersoft"
VALUE "LegalTrademarks", "Muldersoft"
VALUE "Comments", "Released under the GNU General Public License"
END
END
BLOCK "VarFileInfo"
BEGIN
VALUE "Translation", 0x0, 1200
END
END

181
src/TimedExec.cpp
View File

@ -1,6 +1,6 @@
//////////////////////////////////////////////////////////////////////////////////
// Timed Exec - Command-Line Benchmarking Utility
// Copyright (c) 2018 LoRd_MuldeR <mulder2@gmx.de>. Some rights reserved.
// Copyright (c) 2023 LoRd_MuldeR <mulder2@gmx.de>. Some rights reserved.
//
// This program is free software; you can redistribute it and/or
// modify it under the terms of the GNU General Public License
@ -19,6 +19,8 @@
// http://www.gnu.org/licenses/gpl-2.0.txt
//////////////////////////////////////////////////////////////////////////////////
#include "Version.h"
#include <cstdlib>
#include <vector>
#include <algorithm>
@ -34,9 +36,6 @@
#include <ShellAPI.h>
#include <Shlwapi.h>
#define VERSION_MAJOR 1
#define VERSION_MINOR 4
#define DEFAULT_EXEC_LOOPS 5
#define DEFAULT_WARMUP_LOOPS 1
#define DEFAULT_LOGFILE "TimedExec.log"
@ -51,10 +50,44 @@
static HANDLE g_hAbortEvent = NULL;
static volatile bool g_aborted = false;
/* clock for time measurement */
typedef enum
{
CLOCK_WALLCLOCK,
CLOCK_CPU_TOTAL,
CLOCK_CPU_USER,
CLOCK_CPU_KERNEL
}
clock_type_t;
// =============================================================================================================
// INTERNAL FUNCTIONS
// =============================================================================================================
#define _PRINT_CLOCK_TYPE(X) case CLOCK_##X: return #X;
#define _PARSE_CLOCK_TYPE(X) if (!_tcsicmp(name.c_str(), _T(#X))) { clock_type = (CLOCK_##X); return true; }
static bool parseClockType(const tstring &name, clock_type_t &clock_type)
{
_PARSE_CLOCK_TYPE(WALLCLOCK)
_PARSE_CLOCK_TYPE(CPU_TOTAL)
_PARSE_CLOCK_TYPE(CPU_USER)
_PARSE_CLOCK_TYPE(CPU_KERNEL)
return false;
}
static const char *printClockType(const clock_type_t clock_type)
{
switch (clock_type)
{
_PRINT_CLOCK_TYPE(WALLCLOCK)
_PRINT_CLOCK_TYPE(CPU_TOTAL)
_PRINT_CLOCK_TYPE(CPU_USER)
_PRINT_CLOCK_TYPE(CPU_KERNEL)
}
return "N/A";
}
static bool getEnvVariable(const _TCHAR *const name, tstring &value)
{
std::vector<TCHAR> buffer;
@ -81,32 +114,48 @@ static bool getEnvVariable(const _TCHAR *const name, tstring &value)
return false;
}
static LONGLONG getCurrentTime(void)
static ULONGLONG fileTimeToU64(const PFILETIME fileTime)
{
LARGE_INTEGER timeValue;
if(!QueryPerformanceCounter(&timeValue))
{
std::cerr << "\n\nSYSTEM ERROR: Failed to query performance counter!\n" << std::endl;
_exit(EXIT_FAILURE);
}
return timeValue.QuadPart;
ULARGE_INTEGER temp;
temp.HighPart = fileTime->dwHighDateTime;
temp.LowPart = fileTime->dwLowDateTime;
return temp.QuadPart;
}
static LONGLONG getTimerFrequency(void)
static ULONGLONG getTimeElapsed(const ULONGLONG timeStart, const ULONGLONG timeExit)
{
LARGE_INTEGER timeValue;
if(!QueryPerformanceFrequency(&timeValue))
return (timeExit > timeStart) ? (timeExit - timeStart) : 0ULL;
}
static double getProcessTime(const HANDLE hProc, const clock_type_t clock_type)
{
FILETIME timeStart, timeExit, timeKernel, timeUser;
ULONGLONG result = 0ULL;
if (GetProcessTimes(hProc, &timeStart, &timeExit, &timeKernel, &timeUser))
{
std::cerr << "\n\nSYSTEM ERROR: Failed to query performance counter frequency!\n" << std::endl;
_exit(EXIT_FAILURE);
switch (clock_type)
{
case CLOCK_WALLCLOCK:
result = getTimeElapsed(fileTimeToU64(&timeStart), fileTimeToU64(&timeExit));
break;
case CLOCK_CPU_USER:
result = fileTimeToU64(&timeUser);
break;
case CLOCK_CPU_KERNEL:
result = fileTimeToU64(&timeKernel);
break;
case CLOCK_CPU_TOTAL:
result = fileTimeToU64(&timeKernel) + fileTimeToU64(&timeUser);
break;
}
}
return timeValue.QuadPart;
return static_cast<double>(result) / 10000000.0;
}
static long long getCurrentFileSize(FILE *const filePtr)
{
struct _stati64 stats;
if(_fstati64(_fileno(filePtr), &stats) == 0)
if (_fstati64(_fileno(filePtr), &stats) == 0)
{
return stats.st_size;
}
@ -183,7 +232,7 @@ static int initializeCommandLine(tstring &commandLine, tstring &programFile)
int nArgs = 0;
TCHAR **szArglist = CommandLineToArgvW(GetCommandLineW(), &nArgs);
if((szArglist == NULL) || (nArgs < 2))
if ((szArglist == NULL) || (nArgs < 2))
{
return 0;
}
@ -192,7 +241,7 @@ static int initializeCommandLine(tstring &commandLine, tstring &programFile)
{
tstring token;
if(i > 1)
if (i > 1)
{
token = tstring(szArglist[i]);
commandLine += _T(' ');
@ -203,7 +252,7 @@ static int initializeCommandLine(tstring &commandLine, tstring &programFile)
programFile += token;
}
if(token.find(' ') == tstring::npos)
if (token.find(' ') == tstring::npos)
{
commandLine += token;
}
@ -226,7 +275,7 @@ static bool createProcess(const tstring &commandLine, HANDLE &hThrd, HANDLE &hPr
PROCESS_INFORMATION processInfo;
SecureZeroMemory(&processInfo, sizeof(PROCESS_INFORMATION));
if(!CreateProcess(NULL, (LPTSTR)commandLine.c_str(), NULL, NULL, false, HIGH_PRIORITY_CLASS | (suspended ? CREATE_SUSPENDED : 0), NULL, NULL, &startInfo, &processInfo))
if (!CreateProcess(NULL, (LPTSTR)commandLine.c_str(), NULL, NULL, false, HIGH_PRIORITY_CLASS | (suspended ? CREATE_SUSPENDED : 0), NULL, NULL, &startInfo, &processInfo))
{
return false;
}
@ -241,11 +290,11 @@ static bool waitForProcess(const HANDLE &hProc)
{
HANDLE waitHandles[2] = {hProc, g_hAbortEvent};
const DWORD ret = WaitForMultipleObjects(2, &waitHandles[0], FALSE, INFINITE);
if((ret != WAIT_OBJECT_0) && (ret != WAIT_OBJECT_0 + 1))
if ((ret != WAIT_OBJECT_0) && (ret != WAIT_OBJECT_0 + 1))
{
return false;
}
if(ret > WAIT_OBJECT_0)
if (ret > WAIT_OBJECT_0)
{
g_aborted = true;
}
@ -255,7 +304,7 @@ static bool waitForProcess(const HANDLE &hProc)
static int getProcessExitCode(const HANDLE &hProc)
{
DWORD exitCode;
if(GetExitCodeProcess(hProc, &exitCode))
if (GetExitCodeProcess(hProc, &exitCode))
{
return *reinterpret_cast<int*>(&exitCode);
}
@ -308,8 +357,8 @@ static int timedExecMain(int argc, _TCHAR* argv[])
initFmt.copyfmt(std::cerr);
std::cerr << "\n===============================================================================" << std::endl;
std::cerr << "Timed Exec - Benchmarking Utility, Version " << VERSION_MAJOR << '.' << std::setfill('0') << std::setw(2) << VERSION_MINOR << " [" __DATE__ "]" << std::endl;
std::cerr << "Copyright (c) 2018 LoRd_MuldeR <mulder2@gmx.de>. Some rights reserved.\n" << std::endl;
std::cerr << "Timed Exec - Benchmarking Utility, Version " << VERSION_MAJOR << '.' << std::setfill('0') << std::setw(2) << (10 * VERSION_MINOR_HI) + VERSION_MINOR_LO << " [" __DATE__ "]" << std::endl;
std::cerr << "Copyright (c) 2023 LoRd_MuldeR <mulder2@gmx.de>. Some rights reserved.\n" << std::endl;
std::cerr << "This program is free software: you can redistribute it and/or modify" << std::endl;
std::cerr << "it under the terms of the GNU General Public License <http://www.gnu.org/>." << std::endl;
std::cerr << "Note that this program is distributed with ABSOLUTELY NO WARRANTY." << std::endl;
@ -322,7 +371,7 @@ static int timedExecMain(int argc, _TCHAR* argv[])
/* ---------------------------------------------------------- */
tstring commandLine, programFile;
if(initializeCommandLine(commandLine, programFile) < 2)
if (initializeCommandLine(commandLine, programFile) < 2)
{
std::cerr << "Usage:" << std::endl;
std::cerr << " TimedExec.exe <Program.exe> [Arguments]\n" << std::endl;
@ -330,17 +379,18 @@ static int timedExecMain(int argc, _TCHAR* argv[])
std::cerr << " TIMED_EXEC_PASSES - Number of execution passes (default: " << DEFAULT_EXEC_LOOPS << ")" << std::endl;
std::cerr << " TIMED_EXEC_WARMUP_PASSES - Number of warm-up passes (default: " << DEFAULT_WARMUP_LOOPS << ")" << std::endl;
std::cerr << " TIMED_EXEC_LOGFILE - Log-File Name (default: \"" << DEFAULT_LOGFILE << "\")" << std::endl;
std::cerr << " TIMED_EXEC_NO_CHECKS - Set this to *disable* exit code checks\n" << std::endl;
std::cerr << " TIMED_EXEC_NO_CHECKS - Set this to *disable* exit code checks" << std::endl;
std::cerr << " TIMED_EXEC_CLOCK_TYPE - The type of clock used for measurements\n" << std::endl;
return EXIT_FAILURE;
}
if(_taccess(programFile.c_str(), 0) != 0)
if (_taccess(programFile.c_str(), 0) != 0)
{
_ftprintf(stderr, _T("Specified program file could not be found or access denied:\n%s\n\n"), programFile.c_str());
return EXIT_FAILURE;
}
if(!checkBinary(programFile))
if (!checkBinary(programFile))
{
_ftprintf(stderr, _T("Specified file does not look like a valid Win32 executable:\n%s\n\n"), programFile.c_str());
return EXIT_FAILURE;
@ -353,26 +403,35 @@ static int timedExecMain(int argc, _TCHAR* argv[])
tstring logFile(getFullPath(_T(DEFAULT_LOGFILE)));
int maxPasses = DEFAULT_EXEC_LOOPS, maxWarmUpPasses = DEFAULT_WARMUP_LOOPS;
bool checkExitCodes = true;
clock_type_t clock_type = CLOCK_WALLCLOCK;
if(ENABLE_ENV_VARS)
if (ENABLE_ENV_VARS)
{
tstring temp;
if(getEnvVariable(_T("TIMED_EXEC_PASSES"), temp))
if (getEnvVariable(_T("TIMED_EXEC_PASSES"), temp))
{
maxPasses = std::min(SHRT_MAX, std::max(3, _tstoi(temp.c_str())));
}
if(getEnvVariable(_T("TIMED_EXEC_WARMUP_PASSES"), temp))
if (getEnvVariable(_T("TIMED_EXEC_WARMUP_PASSES"), temp))
{
maxWarmUpPasses = std::min(SHRT_MAX, std::max(0, _tstoi(temp.c_str())));
}
if(getEnvVariable(_T("TIMED_EXEC_LOGFILE"), temp))
if (getEnvVariable(_T("TIMED_EXEC_LOGFILE"), temp))
{
logFile = getFullPath(temp.c_str());
}
if(getEnvVariable(_T("TIMED_EXEC_NO_CHECKS"), temp))
if (getEnvVariable(_T("TIMED_EXEC_NO_CHECKS"), temp))
{
checkExitCodes = (_tstoi(temp.c_str()) == 0);
}
if (getEnvVariable(_T("TIMED_EXEC_CLOCK_TYPE"), temp))
{
if (!parseClockType(temp, clock_type))
{
_ftprintf(stderr, _T("Specified clock type \"%s\" is unsupported.\nPlease see the documentation for a list of supported clock types!\n\n"), temp.c_str());
return EXIT_FAILURE;
}
}
}
/* ---------------------------------------------------------- */
@ -383,13 +442,9 @@ static int timedExecMain(int argc, _TCHAR* argv[])
_ftprintf(stderr, _T("Log File:\n%s\n\n"), logFile.c_str());
std::cerr << "Warm-Up / Metering passes: " << maxWarmUpPasses << "x / " << maxPasses << 'x' << std::endl;
const LONGLONG timerFrequency = getTimerFrequency();
if(!SetPriorityClass(GetCurrentProcess(), REALTIME_PRIORITY_CLASS))
if (!SetPriorityClass(GetCurrentProcess(), HIGH_PRIORITY_CLASS))
{
if(!SetPriorityClass(GetCurrentProcess(), HIGH_PRIORITY_CLASS))
{
std::cerr << "\nWARNING: Failed to set process priroity class!" << std::endl;
}
std::cerr << "\nWARNING: Failed to set process priroity class!" << std::endl;
}
std::vector<double> stats_samples(maxPasses, 0.0);
@ -403,7 +458,7 @@ static int timedExecMain(int argc, _TCHAR* argv[])
/* Run Warm-Up Passes */
/* ---------------------------------------------------------- */
for(int pass = 0; pass < maxWarmUpPasses; pass++)
for (int pass = 0; pass < maxWarmUpPasses; pass++)
{
std::cerr << "\n===============================================================================" << std::endl;
if (maxWarmUpPasses > 1) std::cerr << "WARM-UP PASS " << (pass + 1) << " OF " << maxWarmUpPasses << std::endl; else std::cerr << "WARM-UP PASS" << std::endl;
@ -411,26 +466,26 @@ static int timedExecMain(int argc, _TCHAR* argv[])
HANDLE hThrd, hProc;
if(!createProcess(commandLine, hThrd, hProc))
if (!createProcess(commandLine, hThrd, hProc))
{
std::cerr << "\n\nSYSTEM ERROR: Failed to create process!\n" << std::endl;
return EXIT_FAILURE;
}
if(!waitForProcess(hProc))
if (!waitForProcess(hProc))
{
std::cerr << "\n\nSYSTEM ERROR: Failed to wait for process termination!\n" << std::endl;
return EXIT_FAILURE;
}
if(g_aborted)
if (g_aborted)
{
abortedHandlerRoutine(hProc);
return EXIT_FAILURE;
}
const int exitCode = getProcessExitCode(hProc);
if(checkExitCodes && (exitCode != 0))
if (checkExitCodes && (exitCode != 0))
{
std::cerr << "\n\nPROGRAM ERROR: Abnormal program termination detected! (Exit Code: " << exitCode << ")\n" << std::endl;
return EXIT_FAILURE;
@ -456,38 +511,33 @@ static int timedExecMain(int argc, _TCHAR* argv[])
HANDLE hThrd, hProc;
if(!createProcess(commandLine, hThrd, hProc, true))
if (!createProcess(commandLine, hThrd, hProc, true))
{
std::cerr << "\n\nSYSTEM ERROR: Failed to create process!\n" << std::endl;
return EXIT_FAILURE;
}
const LONGLONG timeStart = getCurrentTime();
if(ResumeThread(hThrd) == ((DWORD) -1))
if (ResumeThread(hThrd) == ((DWORD) -1))
{
std::cerr << "\n\nSYSTEM ERROR: Failed to resume child process!\n" << std::endl;
TerminateProcess(hProc, UINT(-1));
return EXIT_FAILURE;
}
if(!waitForProcess(hProc))
if (!waitForProcess(hProc))
{
std::cerr << "\n\nSYSTEM ERROR: Failed to wait for process termination!\n" << std::endl;
return EXIT_FAILURE;
}
const LONGLONG timeFinish = getCurrentTime();
const double execTime = static_cast<double>(timeFinish - timeStart) / static_cast<double>(timerFrequency);
if(g_aborted)
if (g_aborted)
{
abortedHandlerRoutine(hProc);
return EXIT_FAILURE;
}
const int exitCode = getProcessExitCode(hProc);
if(checkExitCodes && (exitCode != 0))
if (checkExitCodes && (exitCode != 0))
{
std::cerr << "\n\nPROGRAM ERROR: Abnormal program termination detected! (Exit Code: " << exitCode << ")\n" << std::endl;
return EXIT_FAILURE;
@ -497,6 +547,8 @@ static int timedExecMain(int argc, _TCHAR* argv[])
std::cerr << "\n>> Process terminated with exit code " << exitCode << '.' << std::endl;
}
const double execTime = getProcessTime(hProc, clock_type);
std::cerr << std::setprecision(3) << std::fixed;
std::cerr << ">> Execution took " << execTime << " seconds." << std::endl;
std::cerr.copyfmt(initFmt);
@ -508,8 +560,8 @@ static int timedExecMain(int argc, _TCHAR* argv[])
stats_samples[pass] = execTime;
// Update slowest/fastest
if(execTime > stats_slowest) stats_slowest = execTime;
if(execTime < stats_fastest) stats_fastest = execTime;
if (execTime > stats_slowest) stats_slowest = execTime;
if (execTime < stats_fastest) stats_fastest = execTime;
// Iterative "online" computation of the mean and the variance
// See http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Online_algorithm for details!
@ -548,6 +600,7 @@ static int timedExecMain(int argc, _TCHAR* argv[])
std::cerr << "Standard Deviation : " << standardDeviation << " seconds" << std::endl;
std::cerr << "Standard Error : " << standardError << " seconds" << std::endl;
std::cerr << "Fastest / Slowest Pass : " << stats_fastest << " / " << stats_slowest << " seconds" << std::endl;
std::cerr << "Active Clock Type : " << printClockType(clock_type) << " (" << clock_type << ')' << std::endl;
std::cerr << "===============================================================================\n" << std::endl;
std::cerr.copyfmt(initFmt);
@ -556,14 +609,14 @@ static int timedExecMain(int argc, _TCHAR* argv[])
/* ---------------------------------------------------------- */
FILE *fLog = NULL;
if(_tfopen_s(&fLog, logFile.c_str(), _T("a+")) == 0)
if (_tfopen_s(&fLog, logFile.c_str(), _T("a+")) == 0)
{
if(getCurrentFileSize(fLog) == 0)
if (getCurrentFileSize(fLog) == 0)
{
_ftprintf_s(fLog, _T("%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n"), _T("Program"), _T("Passes"), _T("Mean Time"), _T("Median Time"), _T("90% Confidence Interval"), _T("95% Confidence Interval"), _T("99% Confidence Interval"), _T("Fastest Pass"), _T("Slowest Pass"), _T("Standard Deviation"), _T("Standard Error"), _T("Command Line"));
}
_ftprintf_s(fLog, _T("%s\t%d\t%f\t%f\t%f\t%f\t%f\t%f\t%f\t%f\t%f\t%s\n"), getFileNameOnly(programFile).c_str(), maxPasses, stats_mean, medianTime, confidenceInterval_90, confidenceInterval_95, confidenceInterval_99, stats_fastest, stats_slowest, standardDeviation, standardError, commandLine.c_str());
if(ferror(fLog) == 0)
if (ferror(fLog) == 0)
{
_ftprintf(stderr, _T("Log file updated successfully.\n\n"));
}
@ -634,7 +687,7 @@ int _tmain(int argc, _TCHAR* argv[])
_set_invalid_parameter_handler(invalidParameterHandler);
#endif // _DEBUG
if(!(g_hAbortEvent = CreateEvent(NULL, TRUE, FALSE, NULL)))
if (!(g_hAbortEvent = CreateEvent(NULL, TRUE, FALSE, NULL)))
{
std::cerr << "\n\nSYSTEM ERROR: Event object could not be created!\n" << std::endl;
return EXIT_FAILURE;

29
src/Version.h Normal file
View File

@ -0,0 +1,29 @@
//////////////////////////////////////////////////////////////////////////////////
// Timed Exec - Command-Line Benchmarking Utility
// Copyright (c) 2023 LoRd_MuldeR <mulder2@gmx.de>. Some rights reserved.
//
// This program is free software; you can redistribute it and/or
// modify it under the terms of the GNU General Public License
// as published by the Free Software Foundation; either version 2
// of the License, or (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program; if not, write to the Free Software
// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
//
// http://www.gnu.org/licenses/gpl-2.0.txt
//////////////////////////////////////////////////////////////////////////////////
#ifndef _INC_TIMED_EXEC_VERSION_H
#define _INC_TIMED_EXEC_VERSION_H
#define VERSION_MAJOR 1
#define VERSION_MINOR_HI 0
#define VERSION_MINOR_LO 5
#endif

12
z_build.bat
View File

@ -108,13 +108,12 @@ echo BEGIN PACKAGING [Release]
echo ---------------------------------------------------------------------
mkdir "%PACK_PATH%"
mkdir "%PACK_PATH%\x64"
copy "%~dp0\bin\Win32\Release\*.exe" "%PACK_PATH%"
copy "%~dp0\bin\x64\.\Release\*.exe" "%PACK_PATH%\x64"
copy "%~dp0\LICENSE.html" "%PACK_PATH%"
copy "%~dp0\bin\Win32\Release\TimedExec.exe" "%PACK_PATH%\TimedExec-x86.exe"
copy "%~dp0\bin\x64\.\Release\TimedExec.exe" "%PACK_PATH%\TimedExec-x64.exe"
copy "%~dp0\LICENSE.html" "%PACK_PATH%"
"%PDOC_PATH%\pandoc.exe" --from markdown_github+pandoc_title_block+header_attributes+implicit_figures+inline_notes --to html5 --toc -N --standalone -H "%~dp0\..\Prerequisites\Pandoc\css\github-pandoc.inc" "README.md" | "%JAVA_HOME%\bin\java.exe" -jar "%~dp0\..\Prerequisites\HTMLCompressor\bin\htmlcompressor-1.5.3.jar" --compress-css -o "%PACK_PATH%\README.html"
"%PDOC_PATH%\pandoc.exe" --from markdown_github+pandoc_title_block+header_attributes+implicit_figures+inline_notes --to html5 --toc -N --metadata title="TimedExec" --standalone -H "%~dp0\..\Prerequisites\Pandoc\css\github-pandoc.inc" "README.md" | "%JAVA_HOME%\bin\java.exe" -jar "%~dp0\..\Prerequisites\HTMLCompressor\bin\htmlcompressor-1.5.3.jar" --compress-css -o "%PACK_PATH%\README.html"
mkdir "%PACK_PATH%\img"
mkdir "%PACK_PATH%\img\timedexec"
@ -125,13 +124,12 @@ REM ///////////////////////////////////////////////////////////////////////////
REM // Compress
REM ///////////////////////////////////////////////////////////////////////////
"%UPX3_PATH%\upx.exe" --best "%PACK_PATH%\*.exe"
"%UPX3_PATH%\upx.exe" --best "%PACK_PATH%\x64\*.exe"
REM ///////////////////////////////////////////////////////////////////////////
REM // Create version tag
REM ///////////////////////////////////////////////////////////////////////////
echo TimedExec> "%PACK_PATH%\BUILD_TAG"
echo Copyright (C) 2018 LoRd_MuldeR ^<MuldeR2@GMX.de^>>> "%PACK_PATH%\BUILD_TAG"
echo Copyright (C) 2023 LoRd_MuldeR ^<MuldeR2@GMX.de^>>> "%PACK_PATH%\BUILD_TAG"
echo.>> "%PACK_PATH%\BUILD_TAG"
echo Built on %ISO_DATE%, at %ISO_TIME%>> "%PACK_PATH%\BUILD_TAG"
echo.>> "%PACK_PATH%\BUILD_TAG"