You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-12 - 08:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Mon Jan 12, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1973929466,8sleep20-21swapper/219:08:072
1962329277,11sleep00-21swapper/019:06:370
1968129042,44sleep10-21swapper/119:07:221
1968328762,9sleep30-21swapper/319:07:253
1997399293,22cyclictest225-21systemd-journal22:29:592
1997399280,3cyclictest5744-21latency_hist23:20:002
1997399280,28cyclictest0-21swapper/223:13:282
1997399280,14cyclictest21107-31find00:07:082
1997399274,22cyclictest12796-21diskmemload23:57:562
1997399272,22cyclictest0-21swapper/221:10:132
1997399270,0cyclictest0-21swapper/221:45:512
1997399263,23cyclictest12796-21diskmemload22:16:392
1997399260,2cyclictest0-21swapper/200:30:162
1997399260,0cyclictest0-21swapper/223:48:552
1996999260,22cyclictest0-21swapper/121:32:101
1996999260,1cyclictest0-21swapper/123:22:051
1997399250,2cyclictest0-21swapper/200:26:242
1997399250,25cyclictest0-21swapper/223:08:512
1997399242,3cyclictest22077-21apache219:20:002
1997399240,23cyclictest0-21swapper/200:36:562
19969992421,2cyclictest22084-21apache200:02:061
1997399232,2cyclictest12672-21kworker/u8:122:33:192
1997399230,2cyclictest26704-21tail19:20:282
1997399230,22cyclictest0-21swapper/200:21:232
1997399230,1cyclictest0-21swapper/222:01:022
1997399230,1cyclictest0-21swapper/221:40:072
1996999230,1cyclictest3752-21cut22:40:141
1996499231,1cyclictest0-21swapper/021:40:130
1997399221,2cyclictest12796-21diskmemload23:22:402
1997399220,3cyclictest0-21swapper/223:33:072
1997399220,22cyclictest0-21swapper/223:41:012
1997399220,22cyclictest0-21swapper/223:41:012
1997399220,22cyclictest0-21swapper/200:18:272
1997399220,21cyclictest0-21swapper/223:25:062
1997399220,1cyclictest0-21swapper/221:25:492
1997399220,0cyclictest0-21swapper/222:49:012
19973992121,0cyclictest0-21swapper/222:10:412
1997399210,2cyclictest12796-21diskmemload22:42:162
1997399210,1cyclictest0-21swapper/222:20:252
1997399210,1cyclictest0-21swapper/221:15:142
1996999210,3cyclictest0-21swapper/123:05:161
1997399203,16cyclictest28934-21tr22:55:132
1997399201,2cyclictest29532-21munin-run19:30:002
1997399201,1cyclictest12796-21diskmemload21:55:172
1997399200,2cyclictest0-21swapper/223:00:232
1997399200,2cyclictest0-21swapper/200:10:292
1997399200,20cyclictest0-21swapper/222:37:062
1997399200,1cyclictest0-21swapper/222:50:312
1997399200,18cyclictest0-21swapper/223:54:382
1997399200,17cyclictest0-21swapper/221:22:502
1996499200,1cyclictest0-21swapper/022:10:120
1997399198,11cyclictest12796-21diskmemload23:37:052
1997399191,18cyclictest0-21swapper/200:03:552
1997399191,12cyclictest0-21swapper/220:30:012
1996999193,12cyclictest0-21swapper/122:00:471
19969991914,3cyclictest0-21swapper/100:24:221
1996499192,1cyclictest12796-21diskmemload23:59:550
19964991916,2cyclictest13-21ksoftirqd/021:14:100
1997599180,17cyclictest0-21swapper/319:50:133
19973991818,0cyclictest0-21swapper/221:05:022
1997399181,1cyclictest30781-21wc20:39:592
1997399181,0cyclictest0-21swapper/219:35:132
1997399180,14cyclictest0-21swapper/222:05:382
1996999181,15cyclictest0-21swapper/100:25:521
1996999180,2cyclictest0-21swapper/100:30:151
1996999180,18cyclictest0-21swapper/123:03:051
1997399171,2cyclictest0-21swapper/221:50:022
1997399171,15cyclictest0-21swapper/221:35:002
1997399170,3cyclictest0-21swapper/221:35:022
1996999171,15cyclictest12796-21diskmemload23:47:551
1996999170,5cyclictest32102-21df_abs23:35:151
1997599160,15cyclictest0-21swapper/321:05:013
1997399163,11cyclictest3052-21apt-get20:50:022
1997399161,1cyclictest14361-21basename20:05:152
1997399161,13cyclictest23601-21latency_hist20:25:002
1997399160,4cyclictest21827-21unixbench-2d19:10:252
19969991614,1cyclictest12796-21diskmemload00:12:501
19969991614,1cyclictest0-21swapper/122:55:401
1996999161,13cyclictest0-21swapper/123:15:281
1996999160,14cyclictest0-21swapper/100:15:041
1996499163,7cyclictest24253-21munin-run21:55:010
1997399154,6cyclictest2104-21taskset19:40:082
1997399152,2cyclictest8941-21sort21:00:152
1997399151,3cyclictest7615-21unixbench_singl20:55:272
19969991512,2cyclictest0-21swapper/122:16:191
19969991510,3cyclictest1704-21cut20:45:161
1996999150,5cyclictest0-21swapper/121:15:271
1996999150,11cyclictest0-21swapper/123:42:151
1996999150,11cyclictest0-21swapper/123:42:151
1996499155,5cyclictest0-21swapper/022:05:140
1996499150,0cyclictest0-21swapper/021:46:320
1997599143,9cyclictest3811-21ssh00:35:163
1997599143,7cyclictest0-21swapper/323:45:593
1997599142,7cyclictest0-21swapper/322:35:133
19975991410,3cyclictest28119-21head19:25:163
1997399147,6cyclictest12723-21cat20:00:182
1997399140,3cyclictest9448-21basename19:55:122
1996999146,7cyclictest14624-21/usr/sbin/munin23:25:211
1996999141,12cyclictest24936-21wc23:30:261
19969991410,3cyclictest6388-21tail19:45:251
19969991410,3cyclictest28244-21wc22:55:001
19969991410,2cyclictest3746-21grep19:40:241
1996999140,4cyclictest225-21systemd-journal00:06:451
1996499143,7cyclictest4076-21/usr/sbin/munin22:40:150
1996499142,6cyclictest30732-21perf20:39:590
19964991411,2cyclictest23958-21ssh23:50:010
19964991410,3cyclictest18968-21tr00:25:150
133222140,0chrt0-21swapper/323:05:133
1997599139,3cyclictest24381-21wc21:35:193
1997599139,3cyclictest0-21swapper/322:14:233
1997599134,5cyclictest24282-21apt-get23:50:123
19975991311,1cyclictest217142sleep300:07:173
19975991310,2cyclictest8373-21cut21:45:173
19975991310,2cyclictest501-21grep23:35:233
19975991310,2cyclictest39-21ksoftirqd/322:59:233
19975991310,2cyclictest18352-21http22:29:593
19975991310,2cyclictest16859-21grep20:10:173
19975991310,2cyclictest1502-21grep23:55:213
1997599130,1cyclictest0-21swapper/322:09:123
1997599130,12cyclictest28462-21cat20:34:593
1997599130,12cyclictest28164-21cron22:55:003
1997399138,4cyclictest7756-21grep19:50:162
1997399138,4cyclictest2755-21taskset20:46:252
1997399136,6cyclictest27173-21sort20:30:182
1997399130,2cyclictest32402-21ls20:40:272
1996999139,3cyclictest24099-21sed21:35:181
1996999139,2cyclictest9255-21grep22:05:131
19969991310,2cyclictest8171-21basename00:00:001
19969991310,2cyclictest28117-21grep19:25:161
19969991310,2cyclictest26492-21grep20:30:141
19969991310,2cyclictest15568-21wc20:05:231
1996499139,3cyclictest30296-21rm21:58:110
1996499139,3cyclictest25006-21grep23:30:260
1996499136,4cyclictest22542-21apt-get21:35:010
1996499136,2cyclictest1037-21gdbus00:06:450
1996499134,7cyclictest19038-21apache_accesses22:30:120
1996499132,7cyclictest25044-21apt-get19:20:140
19964991311,2cyclictest22077-21apache222:16:470
19964991311,1cyclictest11699-21ssh00:20:220
19964991310,2cyclictest4258-21grep20:50:170
19964991310,2cyclictest4237-21cron19:45:000
19964991310,2cyclictest31717-21grep20:40:200
19964991310,2cyclictest31098-21grep23:15:170
19964991310,2cyclictest30580-21grep21:20:150
19964991310,2cyclictest2917-21cron22:39:590
19964991310,2cyclictest24978-21awk20:25:220
19964991310,2cyclictest21251-21sort20:20:010
19964991310,2cyclictest18880-21cat20:15:000
19964991310,2cyclictest15524-21grep23:25:180
1996499130,2cyclictest0-21swapper/019:35:140
1996499130,12cyclictest24840-21cat19:20:000
1997599129,2cyclictest9309-21sed23:40:263
1997599129,2cyclictest9309-21sed23:40:263
1997599129,2cyclictest833-21strings19:35:203
1997599129,2cyclictest31950-21tr21:40:143
1997599129,2cyclictest28283-21chrt21:18:543
1997599129,2cyclictest19752-21basename22:50:013
1997599129,2cyclictest15625-21wc21:30:183
1997599129,2cyclictest1550-21rm22:20:003
1997599129,2cyclictest12220-21grep00:20:273
1997599128,3cyclictest29099-21grep00:30:293
1997599127,1cyclictest0-21swapper/323:10:143
1997599121,2cyclictest21408-21apt-get20:20:123
1997599121,1cyclictest12796-21diskmemload00:02:333
1997599120,9cyclictest0-21swapper/321:26:243
1997599120,11cyclictest19732-21grep20:15:163
1997399128,3cyclictest184762sleep220:11:502
1997399122,2cyclictest225-21systemd-journal19:45:022
1997399121,3cyclictest225-21systemd-journal19:35:012
1997399121,2cyclictest225-21systemd-journal20:16:452
1996999129,2cyclictest6635-21tail20:55:181
1996999129,2cyclictest3352-21grep00:35:141
1996999129,2cyclictest31756-21grep20:40:201
1996999129,2cyclictest28378-21head22:35:191
1996999129,2cyclictest25204-21ls21:55:131
1996999129,2cyclictest22666-21uniq23:10:151
1996999129,2cyclictest21224-21grep21:15:001
1996999129,2cyclictest17100-21cut21:50:141
1996999129,2cyclictest12666-21cut22:45:181
1996999128,3cyclictest32408-21grep19:35:141
1996999128,3cyclictest3233-21cut22:20:191
1996999128,2cyclictest24543-21grep20:25:181
1996999127,4cyclictest10704-21cut21:05:131
1996999123,7cyclictest537-21snmpd21:42:461
19969991211,1cyclictest0-21swapper/122:31:451
1996499129,2cyclictest8026-21systemctl21:00:000
1996499129,2cyclictest6353-21grep23:20:130
1996499129,2cyclictest3855-21grep00:35:170
1996499129,2cyclictest28508-21grep19:25:190
1996499129,2cyclictest22611-21grep21:15:180
1996499129,2cyclictest16653-21cat00:05:000
1996499129,2cyclictest1505-21ssh22:00:180
1996499123,1cyclictest12796-21diskmemload00:30:180
1996499120,3cyclictest0-21swapper/023:44:100
1996499120,3cyclictest0-21swapper/023:44:090
1997599118,2cyclictest8047-21cat23:20:263
1997599118,2cyclictest7603-21sed20:55:273
1997599118,2cyclictest700-21ssh22:00:113
1997599118,2cyclictest6384-21ssh22:22:173
1997599118,2cyclictest6125-21head23:00:203
1997599118,2cyclictest3925-21expr22:40:143
1997599118,2cyclictest31221-21tr20:40:163
1997599118,2cyclictest30667-21taskset20:39:513
1997599118,2cyclictest26737-21grep00:10:163
1997599118,2cyclictest26443-21ssh23:31:273
1997599118,2cyclictest2567-21ssh23:17:313
1997599118,2cyclictest25221-21expr21:55:133
1997599118,2cyclictest24591-21grep20:25:193
1997599118,2cyclictest20092-21grep00:25:233
1997599118,2cyclictest18765-21ssh22:30:013
1997599118,2cyclictest16506-21tr21:50:123
1997599118,2cyclictest14401-21sh23:25:013
1997599118,2cyclictest13254-21ssh21:10:103
1997599118,2cyclictest11978-21tr20:00:143
1997599118,2cyclictest11187-21sed19:55:263
1997599110,10cyclictest2435-21ssh21:22:323
1996999118,2cyclictest6918-21ssh21:25:161
1996999118,2cyclictest29911-21cut19:30:131
1996999118,2cyclictest29287-21tr20:35:181
1996999118,2cyclictest27366-21ssh23:51:171
1996999118,2cyclictest23105-21grep20:20:251
1996999118,2cyclictest20944-21grep19:10:161
1996999118,2cyclictest20724-21head20:15:271
1996999118,2cyclictest2012-21chrt21:22:131
1996999118,2cyclictest10440-21apache_accesses22:25:131
1996999117,3cyclictest11576-21cron20:00:011
1996999115,3cyclictest16629-21apt-get20:10:141
1996999112,4cyclictest225-21systemd-journal19:20:001
19969991110,1cyclictest0-21swapper/122:12:471
1996999110,2cyclictest0-21swapper/121:45:131
1996499119,1cyclictest12796-21diskmemload22:25:590
1996499118,2cyclictest6580-21expr21:25:150
1996499118,2cyclictest3756-21grep00:15:250
1996499118,2cyclictest32591-21ssh22:56:540
1996499118,2cyclictest3247-21grep22:20:190
1996499118,2cyclictest31288-21ssh23:35:030
1996499118,2cyclictest30600-21ls19:30:170
1996499118,2cyclictest21902-21tr23:10:120
1996499118,2cyclictest21673-21grep22:50:220
1996499118,2cyclictest15256-21ssh22:46:300
1996499118,2cyclictest11217-21grep21:05:150
1996499117,3cyclictest22584-21tr20:20:190
1996499115,2cyclictest225-21systemd-journal21:00:000
1996499113,7cyclictest7107-21apt-get19:50:130
1996499112,4cyclictest0-21swapper/000:10:120
1996499110,4cyclictest0-21swapper/023:00:150
1996499110,10cyclictest0-21swapper/023:52:380
1997599108,1cyclictest6253-21cat19:45:243
1997599108,1cyclictest3521-21apache_processe20:50:133
1997599108,1cyclictest2815-21uniq19:40:163
1997599107,2cyclictest3870-21ssh00:15:263
1997599107,2cyclictest30016-21expr19:30:153
1997599107,2cyclictest24250-21date19:15:383
1997599107,2cyclictest20966-21head19:10:163
1997599107,2cyclictest12343-21sed21:05:263
1996999108,1cyclictest9638-21grep21:00:211
1996999108,1cyclictest3337-21dpkg20:50:001
1996999108,1cyclictest268612sleep119:21:291
1996999107,2cyclictest7372-21grep19:50:141
1996999107,2cyclictest13269-21python20:00:261
1996499108,1cyclictest5172-21cat19:45:140
1996499108,1cyclictest15617-21ssh21:30:180
1996499108,1cyclictest14616-21expr20:05:150
1996499108,1cyclictest13246-21ssh23:05:110
1996499107,2cyclictest1696-21sed20:45:160
1996499107,2cyclictest10205-21grep19:55:170
1996499106,3cyclictest12470-21grep20:00:160
199759997,1cyclictest26992-21taskset19:23:103
199759997,1cyclictest16167-21taskset20:07:433
199759997,1cyclictest1115-21ls20:45:133
199649997,1cyclictest27754-21ls20:30:260
199649996,2cyclictest20344-21grep19:10:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional