You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-07 - 05:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun Dec 07, 2025 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1457422530,3sleep30-21swapper/319:05:273
149562240160,25sleep00-21swapper/019:07:460
147802234158,25sleep20-21swapper/219:05:362
148372202164,24sleep10-21swapper/119:06:191
1844221620,4sleep21523499cyclictest21:40:222
1844221620,4sleep21523499cyclictest21:40:222
884521500,5sleep01522399cyclictest22:33:280
226822610,1sleep20-21swapper/223:05:132
167802560,2sleep20-21swapper/219:10:262
15234995249,2cyclictest0-21swapper/200:37:452
138672520,4sleep113871-21unixbench_multi20:15:281
156992460,1sleep00-21swapper/021:35:150
1524099453,26cyclictest86550irq/126-enp1s0-00:16:233
15234994340,2cyclictest0-21swapper/219:21:432
15223994011,13cyclictest25037-21munin-run00:25:020
1524099381,17cyclictest0-21swapper/321:51:523
1524099374,8cyclictest139842sleep320:16:013
1524099363,26cyclictest86850irq/129-enp1s0-00:31:273
1524099341,22cyclictest0-21swapper/300:06:433
15240993312,9cyclictest836-21runrttasks20:47:013
1524099321,3cyclictest0-21swapper/321:26:433
1524099321,15cyclictest0-21swapper/300:27:573
1524099311,11cyclictest20506-21kworker/u8:000:20:213
1524099301,26cyclictest0-21swapper/300:10:153
1524099301,13cyclictest0-21swapper/320:50:153
1522399304,12cyclictest28161-21awk22:05:020
15240992912,1cyclictest0-21swapper/321:55:183
1524099281,2cyclictest86750irq/128-enp1s0-22:05:363
1524099281,10cyclictest0-21swapper/319:22:533
15228992813,12cyclictest21025-21kernelversion23:00:181
1524099271,11cyclictest4204-21users22:20:303
1524099271,11cyclictest0-21swapper/321:36:493
1524099270,11cyclictest0-21swapper/319:33:473
1524099270,11cyclictest0-21swapper/319:33:473
1522399272,13cyclictest29075-21munin-run19:40:020
1524099265,3cyclictest0-21swapper/323:55:303
1524099263,16cyclictest0-21swapper/322:36:003
15240992610,3cyclictest31562-21grep19:45:133
15234992610,4cyclictest0-21swapper/223:40:152
1522899261,17cyclictest0-21swapper/100:15:171
1522399261,12cyclictest29704-21munin-run20:55:020
1522399261,11cyclictest6035-21sshd21:13:440
1522399261,11cyclictest22709-21/usr/sbin/munin19:25:140
1522399261,10cyclictest6498-21ntpq23:40:210
15223992610,3cyclictest86750irq/128-enp1s0-23:36:130
1522399260,12cyclictest0-21swapper/000:20:020
102182260,5sleep01522399cyclictest20:10:170
171092250,4sleep121-21rcuc/100:05:171
1524099259,3cyclictest736-21munin-run23:30:023
1524099259,3cyclictest0-21swapper/322:17:343
1524099254,5cyclictest0-21swapper/323:08:343
1524099251,21cyclictest0-21swapper/323:19:563
1524099251,13cyclictest0-21swapper/321:42:543
1524099251,13cyclictest0-21swapper/321:42:533
1524099251,10cyclictest0-21swapper/322:56:083
1524099251,10cyclictest0-21swapper/320:29:333
1524099250,11cyclictest0-21swapper/320:31:333
1523499251,6cyclictest0-21swapper/221:53:122
1522899254,18cyclictest27813-40dnf22:01:031
1522899252,6cyclictest0-21swapper/122:55:381
1522899251,18cyclictest28855-21sshd19:37:231
1522399253,11cyclictest25270-21/usr/sbin/munin00:25:190
1522399252,3cyclictest12050irq/123-ahci[0021:05:190
200402240,1sleep10-21swapper/121:45:151
1524099249,3cyclictest0-21swapper/323:00:063
1524099247,3cyclictest28242-21latency_hist22:05:023
1524099247,3cyclictest0-21swapper/319:15:103
1524099241,9cyclictest0-21swapper/322:44:143
1524099241,8cyclictest0-21swapper/323:30:083
1524099241,8cyclictest0-21swapper/322:12:563
1524099241,16cyclictest0-21swapper/321:16:013
1524099241,14cyclictest0-21swapper/321:11:273
1524099241,12cyclictest0-21swapper/323:20:573
1524099241,12cyclictest0-21swapper/319:53:573
1524099240,8cyclictest0-21swapper/300:01:203
1523499241,6cyclictest0-21swapper/221:18:272
1523499241,5cyclictest0-21swapper/200:02:052
1522899247,3cyclictest24960-21diskstats23:10:141
15228992417,4cyclictest0-21swapper/119:45:151
15228992415,3cyclictest0-21swapper/122:38:521
1522899241,18cyclictest0-21swapper/123:55:471
1522899241,16cyclictest29923-21memory23:20:201
1522899241,14cyclictest5060-21sshd23:38:131
1522899240,1cyclictest0-21swapper/122:20:141
1522899240,17cyclictest0-21swapper/120:56:441
1522399248,3cyclictest3535-21/usr/sbin/munin19:55:140
1522399241,15cyclictest86550irq/126-enp1s0-22:56:450
1524099238,3cyclictest0-21swapper/320:40:133
1524099237,4cyclictest29173-21kworker/u8:200:38:513
1524099236,8cyclictest0-21swapper/322:50:143
1524099233,3cyclictest86850irq/129-enp1s0-19:42:173
1524099233,2cyclictest86550irq/126-enp1s0-19:12:503
1524099231,9cyclictest0-21swapper/320:12:053
1524099231,18cyclictest0-21swapper/321:20:073
1524099231,17cyclictest86850irq/129-enp1s0-21:47:063
1522899239,3cyclictest0-21swapper/122:41:361
1522899236,4cyclictest780-21rngd22:15:131
1522899232,5cyclictest10187-21diskstats23:50:151
15228992311,4cyclictest9141-21cat22:35:011
1522899231,14cyclictest2904-21sshd23:33:451
1522399231,6cyclictest0-21swapper/022:38:290
1522399231,14cyclictest0-21swapper/020:30:370
1522399231,10cyclictest2632-21cpu22:20:140
1522399231,10cyclictest23979-21cpu21:55:110
114612231,2sleep20-21swapper/223:50:462
99032221,3sleep09908-21crond20:10:020
324552221,2sleep10-21swapper/123:25:241
303442220,4sleep230342-21sh22:10:012
291932221,2sleep10-21swapper/120:50:291
1524099228,3cyclictest0-21swapper/321:33:493
1524099223,3cyclictest0-21swapper/323:50:383
1524099222,11cyclictest0-21swapper/323:45:023
1524099221,5cyclictest0-21swapper/323:10:073
1524099221,12cyclictest0-21swapper/320:00:253
1524099221,11cyclictest0-21swapper/322:33:223
1524099221,10cyclictest0-21swapper/319:37:333
1524099220,14cyclictest0-21swapper/321:06:233
1524099220,10cyclictest0-21swapper/321:03:573
1523499221,7cyclictest0-21swapper/200:11:422
1523499221,4cyclictest0-21swapper/222:23:472
1522899228,4cyclictest24070-21netstat00:20:211
1522899222,17cyclictest12738-21munin-run21:30:001
1522899221,19cyclictest0-21swapper/122:29:301
1522899221,11cyclictest0-21swapper/121:36:251
15228992210,4cyclictest7437-21ntp_states21:15:221
1522899220,15cyclictest0-21swapper/100:33:241
1522399221,8cyclictest0-21swapper/019:41:440
1522399221,8cyclictest0-21swapper/000:31:390
1522399221,6cyclictest0-21swapper/021:52:550
1522399221,13cyclictest0-21swapper/020:21:420
1522399221,10cyclictest0-21swapper/021:45:280
1524099216,3cyclictest0-21swapper/323:35:103
1524099213,4cyclictest5964-21proc_pri22:25:263
1524099211,2cyclictest0-21swapper/323:45:513
1524099211,15cyclictest23568-21latency19:25:193
1524099211,10cyclictest21868-21kworker/u8:219:55:093
1523499211,6cyclictest0-21swapper/221:25:192
1522899217,4cyclictest30575-21memory00:35:181
1522899215,13cyclictest26261-21needreboot00:25:221
1522899212,16cyclictest25942-21munin-run22:00:011
1522899211,5cyclictest18672-21/usr/sbin/munin00:10:241
1522899211,13cyclictest13304-21df21:30:151
1522899211,13cyclictest0-21swapper/123:19:321
1522899210,4cyclictest0-21swapper/100:00:281
1522399216,2cyclictest0-21swapper/019:17:110
1522399215,14cyclictest22496-21idleruntime-cro19:25:010
1522399211,8cyclictest0-21swapper/023:54:450
1522399211,8cyclictest0-21swapper/023:45:400
1522399211,7cyclictest0-21swapper/023:55:220
1522399211,6cyclictest0-21swapper/022:52:210
1522399211,6cyclictest0-21swapper/020:17:400
1522399211,5cyclictest0-21swapper/000:12:520
1522399211,4cyclictest0-21swapper/022:48:520
1522399211,10cyclictest0-21swapper/022:10:320
15223992110,8cyclictest2276-21cat19:50:210
1522399210,15cyclictest0-21swapper/023:02:010
52292201,2sleep10-21swapper/119:56:421
242332201,3sleep10-21swapper/123:08:181
1524099209,3cyclictest21134-21/usr/sbin/munin20:35:163
1524099207,3cyclictest0-21swapper/320:05:173
1524099203,15cyclictest0-21swapper/322:45:123
1523499204,5cyclictest9802-21kworker/2:123:33:112
1523499203,4cyclictest17703-21sendmail_mailst00:05:272
1523499201,6cyclictest0-21swapper/221:23:482
15234992013,4cyclictest31571-21head22:10:232
1522899201,2cyclictest0-21swapper/120:35:141
1522899201,18cyclictest0-21swapper/123:49:081
1522899201,18cyclictest0-21swapper/120:10:211
1522899201,17cyclictest0-21swapper/119:20:411
1522899201,12cyclictest0-21swapper/122:06:461
1522899200,18cyclictest0-21swapper/122:50:221
1522899200,16cyclictest0-21swapper/121:40:561
1522899200,16cyclictest0-21swapper/121:40:561
1522899200,15cyclictest0-21swapper/122:10:481
1522899200,14cyclictest0-21swapper/121:51:561
1522899200,13cyclictest0-21swapper/121:23:401
1522399206,3cyclictest12050irq/123-ahci[0020:50:020
1522399202,2cyclictest8148-21kworker/0:121:21:120
1522399202,17cyclictest81ktimersoftd/021:42:420
1522399202,17cyclictest81ktimersoftd/021:42:420
1522399201,7cyclictest0-21swapper/020:58:390
1522399201,5cyclictest0-21swapper/000:06:010
1522399201,12cyclictest0-21swapper/023:21:570
1522399201,10cyclictest26838-21cpu23:15:160
1522399200,5cyclictest0-21swapper/022:18:440
1524099195,2cyclictest0-21swapper/320:55:173
1523499199,3cyclictest86750irq/128-enp1s0-22:46:362
1523499195,3cyclictest3955-21diskmemload21:34:062
1523499191,8cyclictest18475-21uniq22:55:152
1523499191,6cyclictest0-21swapper/222:35:532
1523499191,6cyclictest0-21swapper/222:03:012
1523499191,5cyclictest0-21swapper/223:18:392
1523499191,5cyclictest0-21swapper/222:29:432
15234991914,3cyclictest20417-21grep20:30:282
1523499191,3cyclictest86550irq/126-enp1s0-22:15:142
15234991913,3cyclictest25075-21pmu-power21:55:212
15234991913,3cyclictest23727-21gltestperf00:20:172
1523499191,11cyclictest0-21swapper/220:45:182
1522899196,3cyclictest0-21swapper/122:49:231
1522899193,14cyclictest21868-21kworker/u8:220:07:161
1522899191,8cyclictest29305-21/usr/sbin/munin19:40:141
15228991916,2cyclictest0-21swapper/119:31:551
15228991916,2cyclictest0-21swapper/119:31:551
1522899191,3cyclictest0-21swapper/119:51:131
1522899191,2cyclictest0-21swapper/123:42:521
1522899191,2cyclictest0-21swapper/120:28:131
1522899191,17cyclictest0-21swapper/121:08:481
1522899191,17cyclictest0-21swapper/120:47:251
1522899191,17cyclictest0-21swapper/120:44:331
1522899191,17cyclictest0-21swapper/120:21:251
1522899191,17cyclictest0-21swapper/120:00:041
1522899191,17cyclictest0-21swapper/119:11:421
1522899190,18cyclictest0-21swapper/119:28:291
1522399197,3cyclictest26549-21sshd23:14:450
1522399195,11cyclictest14942-21munin-run21:35:010
1522399191,4cyclictest22205-21ntp_states20:35:230
1522399191,15cyclictest6274-21grep22:25:290
1522399191,11cyclictest0-21swapper/019:30:200
1522399191,11cyclictest0-21swapper/019:30:200
1522399191,10cyclictest6840-21pmu-power20:00:250
292752181,3sleep20-21swapper/200:33:092
219032180,2sleep20-21swapper/223:01:362
1524099184,3cyclictest0-21swapper/320:23:413
1523499185,4cyclictest30155-21awk23:20:232
1523499183,12cyclictest9921-21turbostat20:10:012
1523499181,8cyclictest0-21swapper/223:49:062
1523499181,7cyclictest16507-21latency_hist20:25:002
1523499181,6cyclictest0-21swapper/223:38:042
1523499181,6cyclictest0-21swapper/223:12:272
1523499181,4cyclictest0-21swapper/223:56:072
1523499181,4cyclictest0-21swapper/222:43:542
15234991813,3cyclictest26467-21sed19:30:272
15234991813,3cyclictest26467-21sed19:30:272
15234991813,3cyclictest1223-21grep21:00:272
15234991812,3cyclictest12671-21expr20:15:142
1523499181,10cyclictest15400-21cstates21:35:122
1523499181,10cyclictest0-21swapper/200:25:172
1522899187,3cyclictest19771-21kernelversion20:30:211
1522899184,2cyclictest0-21swapper/121:10:161
1522899181,16cyclictest0-21swapper/121:00:491
1522899181,16cyclictest0-21swapper/119:16:031
1522399186,3cyclictest0-21swapper/021:15:160
1522399181,8cyclictest0-21swapper/019:11:430
1522399181,4cyclictest0-21swapper/000:00:250
1522399181,3cyclictest12212-21irqstats22:40:200
1522399181,2cyclictest0-21swapper/021:25:260
15223991812,3cyclictest24402-21ntp_states20:40:260
15223991811,4cyclictest28462-21cat22:05:140
1522399181,10cyclictest0-21swapper/000:35:150
290062171,2sleep20-21swapper/219:39:202
185792171,2sleep20-21swapper/220:28:292
1523499178,3cyclictest0-21swapper/222:30:272
1523499178,2cyclictest0-21swapper/219:45:012
1523499174,4cyclictest0-21swapper/220:56:362
1523499172,5cyclictest7718-21munin-run20:05:012
1523499171,7cyclictest0-21swapper/222:52:202
1523499171,7cyclictest0-21swapper/221:46:422
1523499171,5cyclictest0-21swapper/200:15:092
1523499171,3cyclictest0-21swapper/220:50:282
15234991712,3cyclictest6160-21idleruntime-cro21:15:012
15234991712,3cyclictest31501-21/usr/sbin/munin19:45:182
15234991712,3cyclictest31490-21expr23:25:112
15234991712,3cyclictest24306-21sed20:40:222
15234991712,3cyclictest22709-21/usr/sbin/munin19:25:162
1522399172,3cyclictest86750irq/128-enp1s0-23:30:160
1522399171,8cyclictest0-21swapper/021:00:180
1522399171,6cyclictest31501-21/usr/sbin/munin19:45:240
1522399171,4cyclictest0-21swapper/023:05:440
1522399171,2cyclictest0-21swapper/023:29:320
1522399171,13cyclictest17656-21memory20:25:200
1523499162,6cyclictest11638-21unixbench_multi20:10:292
1523499161,7cyclictest2158-21cstates21:05:142
1523499161,7cyclictest0-21swapper/220:35:122
15234991611,3cyclictest4514-21needreboot19:55:202
15234991610,3cyclictest18298-21basename19:15:052
1523499151,7cyclictest0-21swapper/219:54:512
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional