You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-01 - 13:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Wed Apr 01, 2026 00:43:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
949521140,0sleep30-21swapper/300:24:253
1527829683,9sleep20-21swapper/219:07:042
1519029275,12sleep10-21swapper/119:05:551
1548829066,8sleep00-21swapper/019:09:480
1548128067,9sleep30-21swapper/319:09:423
1559699300,29cyclictest0-21swapper/200:14:322
1559699290,28cyclictest0-21swapper/222:51:022
15599992723,3cyclictest12948-21kworker/3:221:25:143
1559699270,26cyclictest0-21swapper/223:01:292
1559699270,1cyclictest0-21swapper/221:45:072
15596992525,0cyclictest0-21swapper/223:52:192
1559999241,23cyclictest12948-21kworker/3:223:34:543
1559699240,23cyclictest0-21swapper/200:29:032
15594992422,2cyclictest25-21ksoftirqd/100:36:031
15594992422,2cyclictest25-21ksoftirqd/100:36:031
1559999230,16cyclictest0-21swapper/322:50:023
1559699230,5cyclictest0-21swapper/200:30:122
1559499230,16cyclictest0-21swapper/122:17:571
1558999230,1cyclictest0-21swapper/000:10:590
1559999220,21cyclictest12948-21kworker/3:221:13:293
1559499221,19cyclictest26333-21wc19:30:191
15599992118,3cyclictest15168-21kworker/3:000:36:433
15599992118,3cyclictest15168-21kworker/3:000:36:433
15599992117,4cyclictest12948-21kworker/3:223:04:543
1559499212,1cyclictest8121-21diskmemload23:25:301
1559499210,9cyclictest0-21swapper/100:10:091
15599992018,2cyclictest12948-21kworker/3:223:26:233
15599992018,2cyclictest12948-21kworker/3:221:22:533
15599992018,2cyclictest12948-21kworker/3:221:22:533
15599992017,2cyclictest39-21ksoftirqd/321:02:173
15596992020,0cyclictest0-21swapper/223:27:412
1559699200,20cyclictest0-21swapper/222:16:272
15589992017,2cyclictest0-21swapper/000:03:000
15596991915,3cyclictest15197-31find00:09:082
1559699190,19cyclictest0-21swapper/222:34:132
1559699190,16cyclictest0-21swapper/223:36:072
1558999190,1cyclictest0-21swapper/023:16:370
1559999182,14cyclictest0-21swapper/300:09:583
15599991813,4cyclictest477-21rs:main3
15599991813,3cyclictest16515-21sh20:19:593
1559499180,3cyclictest0-21swapper/121:51:541
1559499180,1cyclictest0-21swapper/121:34:561
1558999186,9cyclictest1-21systemd21:04:550
1558999180,17cyclictest0-21swapper/023:01:280
15596991714,1cyclictest0-21swapper/200:22:542
1559699170,16cyclictest0-21swapper/222:08:002
1559999161,2cyclictest0-21swapper/322:55:523
1559699161,13cyclictest0-21swapper/222:13:092
1559699160,16cyclictest0-21swapper/222:28:392
1559499164,10cyclictest25-21ksoftirqd/122:25:171
1559499160,3cyclictest25-21ksoftirqd/123:11:281
1559499160,3cyclictest25-21ksoftirqd/123:11:281
1559499160,2cyclictest0-21swapper/121:49:111
1559499160,16cyclictest0-21swapper/122:54:291
215332150,0chrt0-21swapper/123:32:211
1559999153,8cyclictest7019-21cut22:45:163
1559999152,8cyclictest225-21systemd-journal22:25:213
1559699150,1cyclictest0-21swapper/221:27:012
1559699150,13cyclictest0-21swapper/223:40:562
15594991513,2cyclictest0-21swapper/122:20:041
15594991511,3cyclictest19921-21grep19:15:271
1559499150,3cyclictest0-21swapper/122:36:121
1559999149,3cyclictest19150-21uname00:10:153
15599991412,1cyclictest39-21ksoftirqd/321:06:063
1559699147,2cyclictest20762-21grep21:55:232
15596991413,1cyclictest0-21swapper/200:16:162
1559699141,13cyclictest0-21swapper/200:00:152
1559699140,4cyclictest0-21swapper/221:38:202
1559699140,14cyclictest0-21swapper/223:14:192
1559699140,14cyclictest0-21swapper/223:14:182
1559699140,14cyclictest0-21swapper/222:21:532
1559699140,12cyclictest0-21swapper/223:24:552
1558999142,8cyclictest4824-21apt-get19:55:150
15589991411,2cyclictest26102-21grep21:20:160
15589991411,2cyclictest26102-21grep21:20:160
15589991410,3cyclictest17847-21grep23:30:160
15589991410,3cyclictest16521-21cron21:15:010
15589991410,2cyclictest1933-21grep20:55:170
15589991410,2cyclictest13-21ksoftirqd/023:51:050
1558999140,8cyclictest23834-21/usr/sbin/munin22:55:180
1558999140,2cyclictest0-21swapper/023:41:340
1558999140,0cyclictest0-21swapper/000:29:090
1559999139,3cyclictest26557-21grep20:40:123
1559999139,3cyclictest21025-21wc21:55:273
1559999139,3cyclictest11666-21grep00:25:183
1559999139,2cyclictest537-21snmpd23:06:503
1559999139,2cyclictest537-21snmpd23:06:503
1559999139,2cyclictest3915-21ls21:45:213
1559999139,2cyclictest21307-21sort22:35:003
15599991311,1cyclictest0-21swapper/323:15:533
15599991310,2cyclictest6935-21ls20:00:003
15599991310,2cyclictest418-21grep23:20:153
15599991310,2cyclictest2545-21grep19:50:133
1559999130,1cyclictest0-21swapper/300:15:583
1559999130,0cyclictest0-21swapper/323:39:413
1559699139,3cyclictest31711-21grep22:40:202
15596991313,0cyclictest0-21swapper/223:55:552
15596991310,2cyclictest8160-21grep22:45:252
15596991310,2cyclictest30509-21grep20:45:252
15596991310,2cyclictest27990-21tr19:35:152
15596991310,2cyclictest24862-21cron21:20:002
1559499139,2cyclictest26000-21rm21:20:161
1559499139,2cyclictest26000-21rm21:20:151
1559499138,2cyclictest25088-21munin-plugin-st19:30:001
1559499138,2cyclictest1073-21grep23:20:191
1559499131,1cyclictest8121-21diskmemload00:29:411
15594991310,2cyclictest26289-21ssh00:34:221
15594991310,2cyclictest26243-21latency_hist20:40:011
1559499130,2cyclictest0-21swapper/100:23:471
1559499130,1cyclictest0-21swapper/122:05:141
1559499130,13cyclictest0-21swapper/100:18:091
1559499130,12cyclictest19434-21grep23:50:191
1559499130,11cyclictest6702-21head21:05:141
1558999139,3cyclictest28303-21cat23:55:260
1558999139,3cyclictest1666-21grep21:25:140
1558999139,2cyclictest14438-21chrt22:50:000
1558999132,9cyclictest8121-21diskmemload00:35:550
1558999132,9cyclictest8121-21diskmemload00:35:540
1558999131,7cyclictest22729-21awk19:25:000
15589991310,2cyclictest29819-21cron19:40:000
15589991310,2cyclictest19678-21grep21:55:170
15589991310,2cyclictest17663-21cron23:49:590
15589991310,2cyclictest15159-21grep20:15:160
15589991310,2cyclictest1009-21grep23:20:170
1558999130,2cyclictest0-21swapper/022:21:390
1558999130,2cyclictest0-21swapper/020:10:130
1559999129,2cyclictest4582-21grep22:05:183
1559999129,2cyclictest25835-21grep19:30:153
1559999129,2cyclictest25736-21grep20:35:253
1559999129,2cyclictest24890-21perf21:20:003
1559999129,2cyclictest21318-21grep22:15:143
1559999129,2cyclictest12710-21head20:10:173
1559999128,3cyclictest28015-21cut22:00:143
1559999128,2cyclictest0-21swapper/323:44:573
1559999126,5cyclictest1370-21apache221:41:023
1559999121,7cyclictest8121-21diskmemload23:10:363
1559999121,7cyclictest8121-21diskmemload23:10:363
1559999121,7cyclictest0-21swapper/323:49:113
1559999120,0cyclictest0-21swapper/319:10:143
1559699129,2cyclictest8145-21uniq23:05:162
1559699129,2cyclictest8145-21uniq23:05:162
1559699129,2cyclictest26249-21basename20:40:012
1559699129,2cyclictest21381-21head19:20:192
1559699129,2cyclictest16935-21grep20:20:142
1559699129,2cyclictest16569-21grep19:10:182
1559699128,3cyclictest28062-21tr22:00:142
1559699128,2cyclictest32369-21wc23:20:002
1559699128,2cyclictest25962-21grep19:30:162
1559699128,2cyclictest13608-21grep20:10:262
1559699121,10cyclictest8121-21diskmemload21:42:352
1559699120,1cyclictest0-21swapper/221:32:022
1559699120,11cyclictest31768-21grep19:40:262
1559699120,0cyclictest0-21swapper/221:08:292
1559499129,2cyclictest9238-21sh21:30:001
1559499129,2cyclictest2944-21grep20:55:271
1559499129,2cyclictest21661-21wc19:20:211
1559499129,2cyclictest1667-21ssh23:40:161
1559499129,2cyclictest10346-21sed23:45:151
1559499128,3cyclictest30220-21wc20:45:221
1559499128,3cyclictest27387-21grep23:55:171
1559499128,3cyclictest26447-21ssh23:35:171
1559499128,3cyclictest21612-21cut20:30:131
1559499128,3cyclictest18117-21fgrep20:20:221
1559499128,2cyclictest31001-21dump-pmu-power23:00:001
1559499125,6cyclictest7626-21munin-node21:13:271
1559499124,8cyclictest8121-21diskmemload23:08:371
1559499124,8cyclictest8121-21diskmemload23:08:371
1559499124,4cyclictest225-21systemd-journal20:14:591
1559499120,1cyclictest0-21swapper/122:44:491
1559499120,1cyclictest0-21swapper/121:15:251
1558999129,2cyclictest9462-21grep23:25:170
1558999129,2cyclictest29178-21tr20:45:140
1558999129,2cyclictest20614-21sed00:30:210
1558999128,3cyclictest5576-21ssh22:25:160
1558999128,3cyclictest16341-21ssh00:09:230
1558999128,3cyclictest13463-21ssh22:10:210
1558999128,2cyclictest19090-21cat20:25:000
1558999120,11cyclictest0-21swapper/022:32:440
1559999119,1cyclictest14894-21apache221:30:123
1559999118,2cyclictest7838-21grep20:00:153
1559999118,2cyclictest778-21ssh22:22:123
1559999118,2cyclictest4913-21ssh00:00:463
1559999118,2cyclictest31520-21grep20:50:133
1559999118,2cyclictest28473-21sed23:55:273
1559999118,2cyclictest23666-21tune2fs19:25:183
1559999118,2cyclictest21680-21cut20:30:133
1559999118,2cyclictest19553-21grep23:50:203
1559999118,2cyclictest19309-21grep20:25:153
1559999118,2cyclictest19168-21basename00:30:123
1559999118,2cyclictest14952-21ssh22:11:033
1559999118,2cyclictest10674-21ssh21:50:053
1559999117,3cyclictest18246-21wget21:35:133
1559699118,2cyclictest749-21chrt20:52:332
1559699118,2cyclictest7490-21grep20:00:142
1559699118,2cyclictest6985-21cat20:00:002
1559699118,2cyclictest4341-21grep21:00:192
1559699118,2cyclictest32764-21tr19:45:132
1559699118,2cyclictest25175-21taskset21:20:042
1559699118,2cyclictest25175-21taskset21:20:032
1559699118,2cyclictest23162-21rm22:55:132
1559699118,2cyclictest22840-21hwlatdetect22:35:172
1559699118,2cyclictest2099-21if_enp2s020:55:192
1559699118,2cyclictest18418-21wc19:15:152
1559699118,2cyclictest12319-21head21:50:222
1559699113,4cyclictest225-21systemd-journal19:55:012
1559699110,1cyclictest0-21swapper/223:31:002
1559699110,1cyclictest0-21swapper/221:13:142
1559499119,1cyclictest12405-21grep22:10:141
1559499118,2cyclictest8606-21python20:00:251
1559499118,2cyclictest508-21grep20:50:251
1559499118,2cyclictest4881-21cut19:55:161
1559499118,2cyclictest4126-21grep19:50:261
1559499118,2cyclictest32696-21ls23:00:211
1559499118,2cyclictest29479-21ssh21:41:091
1559499118,2cyclictest27221-21dump-pmu-power21:59:591
1559499118,2cyclictest2512-21seq00:00:131
1559499118,2cyclictest24940-21ssh23:15:151
1559499118,2cyclictest20755-21sh22:34:381
1559499118,2cyclictest1693-21ssh22:03:501
1559499118,2cyclictest10299-21grep20:05:181
1559499115,2cyclictest225-21systemd-journal20:45:001
1559499114,6cyclictest20786-21grep20:25:271
1559499114,3cyclictest225-21systemd-journal20:19:591
1559499110,1cyclictest0-21swapper/100:09:121
1558999118,2cyclictest9873-21wget21:30:150
1558999118,2cyclictest9863-21ssh22:46:390
1558999118,2cyclictest8912-21sed20:00:250
1558999118,2cyclictest637-21taskset20:51:070
1558999118,2cyclictest5166-21ssh22:05:240
1558999118,2cyclictest328-21ssh23:39:380
1558999118,2cyclictest32484-21apache_processe19:45:110
1558999118,2cyclictest30198-21chrt22:40:070
1558999118,2cyclictest28887-21sh22:00:180
1558999118,2cyclictest27043-21ssh00:15:160
1558999118,2cyclictest26512-21taskset21:40:110
1558999118,2cyclictest24390-21grep20:35:130
1558999118,2cyclictest2391-21ssh00:20:120
1558999118,2cyclictest21874-21ssh22:35:100
1558999118,2cyclictest21782-21ntpq22:15:180
1558999118,2cyclictest20340-21basename19:20:000
1558999118,2cyclictest19274-21wget20:25:160
1558999118,2cyclictest15871-21ssh23:10:120
1558999118,2cyclictest15871-21ssh23:10:110
1558999118,2cyclictest10479-21ssh23:06:130
1558999118,2cyclictest10479-21ssh23:06:130
1558999118,2cyclictest10049-21ssh21:49:450
1558999114,2cyclictest26057-21basename21:40:000
1558999110,2cyclictest0-21swapper/021:53:490
1558999110,10cyclictest17726-21grep21:15:170
1559999108,1cyclictest295392sleep319:36:263
1559999108,1cyclictest22855-21sort22:35:183
1559999108,1cyclictest20161-21taskset19:18:113
1559999108,1cyclictest1640-21wc19:45:243
1559999108,1cyclictest10410-21ls20:05:183
1559999107,2cyclictest17196-21expr20:20:163
1559999106,3cyclictest30214-21grep19:40:143
1559999106,3cyclictest21818-21ls19:20:233
1559999100,9cyclictest30112sleep320:56:003
1559699108,2cyclictest30142-21ssh00:35:502
1559699108,2cyclictest30142-21ssh00:35:502
1559699108,1cyclictest9770-21apt23:45:122
1559699108,1cyclictest26893-21ls20:40:132
1559699108,1cyclictest25201-21sed19:30:012
1559699108,1cyclictest19533-21cat20:25:162
1559699108,1cyclictest16333-21chrt20:17:512
1559699107,2cyclictest23163-21grep20:30:252
1559699107,2cyclictest11483-21taskset20:07:252
1559499108,1cyclictest6718-21expr22:45:141
1559499108,1cyclictest5683-21taskset21:04:311
1559499108,1cyclictest28178-21expr19:35:151
1559499108,1cyclictest20261-21ssh21:35:301
1559499108,1cyclictest16638-21idleruntime19:10:181
1559499107,2cyclictest31373-21ps19:40:211
1559499106,1cyclictest834-21head19:45:161
1558999108,1cyclictest9819-21cut20:05:150
1558999108,1cyclictest6101-21grep21:05:120
1558999108,1cyclictest30683-21taskset19:40:160
1558999108,1cyclictest26655-21gzip19:30:220
1558999108,1cyclictest176762sleep019:11:500
1558999107,2cyclictest4424-21chrt19:53:290
1558999107,2cyclictest27232-21expr20:40:150
1558999107,2cyclictest22973-21chrt19:25:120
1558999107,2cyclictest21963-21expr20:30:140
155999997,1cyclictest307112sleep320:46:223
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional