You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-26 - 12:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Thu Mar 26, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1411229542,48sleep30-21swapper/319:07:213
1413729077,9sleep00-21swapper/019:07:380
1409128965,6sleep20-21swapper/219:07:042
1419327557,14sleep10-21swapper/119:08:231
320502700,0sleep20-21swapper/223:39:112
1441299280,1cyclictest0-21swapper/322:22:453
1441099280,27cyclictest0-21swapper/222:55:012
1441099270,26cyclictest0-21swapper/223:02:592
1441099260,23cyclictest0-21swapper/223:12:082
1441099250,22cyclictest0-21swapper/223:58:102
1441299242,0cyclictest0-21swapper/321:51:563
1440299234,13cyclictest0-21swapper/023:15:120
1441099220,22cyclictest0-21swapper/200:23:552
1441099220,21cyclictest0-21swapper/223:30:332
1440599221,3cyclictest20799-21cut20:30:151
1440599220,21cyclictest0-21swapper/122:13:501
14412992118,2cyclictest39-21ksoftirqd/300:35:313
1441299210,0cyclictest0-21swapper/323:28:313
1441099210,21cyclictest0-21swapper/222:50:472
1440599210,3cyclictest17187-21appstreamcli19:15:001
14402992120,1cyclictest0-21swapper/021:19:200
1440299210,1cyclictest0-21swapper/000:24:370
1441099200,20cyclictest0-21swapper/221:19:392
1441099200,20cyclictest0-21swapper/200:29:072
1441099200,19cyclictest0-21swapper/200:02:022
1441099200,17cyclictest0-21swapper/200:12:542
1440599200,1cyclictest7448-21cpuspeed_turbos23:05:141
1440299200,1cyclictest0-21swapper/021:10:130
1441299190,19cyclictest0-21swapper/323:37:493
1441299190,16cyclictest0-21swapper/320:35:133
1441099190,2cyclictest0-21swapper/223:22:542
1441099190,18cyclictest0-21swapper/221:54:232
1441099190,17cyclictest0-21swapper/221:48:002
1440599190,15cyclictest0-21swapper/123:39:341
14412991817,1cyclictest29279-21kworker/3:100:00:333
14412991817,1cyclictest0-21swapper/322:05:113
1441299180,2cyclictest0-21swapper/321:34:203
1441299180,1cyclictest0-21swapper/321:23:233
1441099180,18cyclictest0-21swapper/221:33:342
1441099180,18cyclictest0-21swapper/200:33:132
1441099180,17cyclictest0-21swapper/222:20:582
14405991816,2cyclictest38150irq/131-snd_hda22:00:381
14405991816,2cyclictest38150irq/131-snd_hda22:00:381
14405991811,3cyclictest0-21swapper/119:10:131
14402991813,4cyclictest564-21apache_processe21:25:130
1440299180,0cyclictest0-21swapper/021:33:020
1441099170,16cyclictest0-21swapper/222:31:292
1440299170,17cyclictest0-21swapper/021:51:150
1441299168,5cyclictest0-21swapper/322:51:253
14412991615,1cyclictest4526-21kworker/3:222:26:233
14412991614,1cyclictest0-21swapper/323:45:163
1441099160,0cyclictest0-21swapper/223:45:132
1441099160,0cyclictest0-21swapper/221:27:212
14405991612,4cyclictest25-21ksoftirqd/122:29:061
1440599160,1cyclictest0-21swapper/123:10:321
1440599160,16cyclictest7199-21diskmemload00:17:441
1440599160,0cyclictest0-21swapper/100:24:201
1440299165,6cyclictest10795-21munin-run20:10:000
1440299162,5cyclictest225-21systemd-journal20:30:000
16282152,1sleep2141rcu_preempt22:42:182
14412991515,0cyclictest0-21swapper/323:20:053
14412991514,1cyclictest0-21swapper/322:12:373
14412991512,2cyclictest18866-21chrt00:09:463
1441299150,14cyclictest0-21swapper/323:55:323
1441099151,13cyclictest3106-21wc23:40:252
14410991510,3cyclictest21697-21sort22:15:262
1441099150,15cyclictest0-21swapper/222:49:472
1441099150,14cyclictest0-21swapper/200:18:582
1440599159,5cyclictest13763-21chrt21:13:361
14405991514,1cyclictest0-21swapper/122:54:531
14405991513,1cyclictest2270-21ssh23:40:171
14405991513,1cyclictest0-21swapper/122:40:291
1440599151,1cyclictest7199-21diskmemload22:19:221
14405991510,4cyclictest38150irq/131-snd_hda23:15:181
1440599150,8cyclictest38150irq/131-snd_hda20:15:141
1440599150,2cyclictest0-21swapper/123:22:051
1440599150,13cyclictest7199-21diskmemload00:11:431
1440299155,8cyclictest27872-21apt22:20:000
1440299154,5cyclictest19712-21apt-get00:10:140
1440299151,7cyclictest15269-21ssh23:29:110
1440299151,10cyclictest31576-21apache_processe19:45:110
1440299150,15cyclictest0-21swapper/022:44:000
14412991412,1cyclictest0-21swapper/323:32:373
1441299141,11cyclictest0-21swapper/321:39:173
1441299140,1cyclictest0-21swapper/300:20:593
14410991411,2cyclictest1825-21wc20:55:262
14410991411,2cyclictest0-21swapper/200:35:152
14410991410,2cyclictest22943-21wc22:35:262
1441099140,13cyclictest0-21swapper/222:07:042
1441099140,12cyclictest0-21swapper/222:14:162
1440599148,5cyclictest0-21swapper/123:51:411
1440599148,5cyclictest0-21swapper/123:51:411
14405991410,3cyclictest17683-21grep23:30:161
14405991410,3cyclictest13082-21grep00:05:261
14405991410,2cyclictest19162-21wc20:25:191
1440599140,2cyclictest0-21swapper/122:45:461
1440599140,14cyclictest0-21swapper/123:59:401
1440599140,14cyclictest0-21swapper/122:31:451
1440299146,7cyclictest10807-21munin-run22:10:000
1440299144,5cyclictest18682-21wget21:55:150
1440299141,7cyclictest2220-21munin-run00:00:000
1440299141,0cyclictest0-21swapper/000:08:190
1440299140,12cyclictest7067-21grep20:00:170
92162130,1sleep20-21swapper/223:05:302
1441299139,3cyclictest4140-21grep19:55:153
1441299139,3cyclictest30019-21tr22:40:193
1441299139,3cyclictest15027-21cron23:10:003
14412991311,2cyclictest0-21swapper/300:27:223
14412991310,2cyclictest31224-21ssh22:02:133
14412991310,2cyclictest31224-21ssh22:02:123
14412991310,2cyclictest29487-21grep19:40:143
14412991310,2cyclictest23830-21ssh22:55:183
14412991310,2cyclictest21201-21gzip00:10:233
14412991310,2cyclictest1997-21ssh21:45:143
14412991310,2cyclictest16300-21cut23:10:163
14412991310,2cyclictest16078-21grep20:20:143
14412991310,2cyclictest1198-21sleep21:45:003
1441299130,11cyclictest16926-21dpkg19:15:003
1441099139,3cyclictest25582-21tr20:40:132
14410991310,2cyclictest5847-21wc22:25:212
14410991310,2cyclictest5137-21tr19:55:222
14410991310,2cyclictest25793-21ssh21:20:202
14410991310,2cyclictest25142-21cut19:30:172
14410991310,2cyclictest23651-21grep19:25:252
14410991310,2cyclictest14123-21grep20:15:182
1441099130,1cyclictest0-21swapper/223:15:352
1441099130,13cyclictest0-21swapper/223:53:252
1441099130,13cyclictest0-21swapper/223:53:242
1441099130,13cyclictest0-21swapper/221:55:322
1440599139,3cyclictest29422-21grep20:45:241
1440599139,2cyclictest10909-21wc22:10:001
1440599138,4cyclictest30868-21ssh21:23:461
1440599136,6cyclictest25446-21munin-node21:50:011
1440599135,5cyclictest587-21apt-get21:25:141
1440599134,8cyclictest22589-21grep22:35:221
14405991313,0cyclictest0-21swapper/100:36:251
1440599130,11cyclictest0-21swapper/121:55:021
14402991310,2cyclictest29910-21cat20:50:010
14402991310,2cyclictest27576-21sed19:35:180
14402991310,2cyclictest2310-21ssh21:45:140
14402991310,2cyclictest1177-21cron21:44:590
14402991310,2cyclictest0-21swapper/022:53:230
1440299130,13cyclictest0-21swapper/021:21:340
1441299129,2cyclictest8456-21sh21:30:003
1441299129,2cyclictest7972-21sed20:00:263
1441299129,2cyclictest7192-21apt21:10:003
1441299129,2cyclictest6784-21wc23:05:003
1441299129,2cyclictest341-21head19:45:203
1441299129,2cyclictest31887-21cron23:20:003
1441299129,2cyclictest30823-21sort20:50:163
1441299129,2cyclictest29313-21cat20:45:233
1441299129,2cyclictest2839-21ls23:40:213
1441299129,2cyclictest19417-21ls20:25:223
1441299129,2cyclictest16469-21cstates21:15:153
1441299129,2cyclictest12442sleep320:55:193
1441299128,3cyclictest20721-21grep19:20:203
1441299128,3cyclictest20147-21grep23:50:253
1441299128,3cyclictest20147-21grep23:50:253
1441299128,3cyclictest1801-21grep19:50:123
1441299128,3cyclictest13661-21ls22:30:163
1441099129,2cyclictest9210-21tail23:25:172
1441099129,2cyclictest6732-21grep21:05:252
1441099129,2cyclictest31818-21grep20:50:262
1441099129,2cyclictest28437-21wc22:00:222
1441099129,2cyclictest28437-21wc22:00:222
1441099129,2cyclictest2621-21cut19:50:192
1441099129,2cyclictest20840-21tr20:30:152
1441099129,2cyclictest19869-21tr19:20:152
1441099128,3cyclictest19689-21sort20:25:262
1440599129,2cyclictest31383-21cat19:45:001
1440599129,2cyclictest29854-21wc22:20:191
1440599129,2cyclictest29114-21chrt22:58:511
1440599129,2cyclictest21227-21ssh00:30:151
1440599129,2cyclictest16553-21head20:20:171
1440599126,5cyclictest0-21swapper/121:15:131
1440599123,9cyclictest25446-21munin-node21:48:251
1440599122,7cyclictest38150irq/131-snd_hda00:00:251
1440599120,11cyclictest3812-21cut21:00:191
1440299129,2cyclictest2949-21wc19:50:240
1440299129,2cyclictest26150-21idleruntime-cro23:55:000
1440299129,2cyclictest26150-21idleruntime-cro23:55:000
1440299129,2cyclictest2413-21cron21:00:000
1440299129,2cyclictest18205-21head21:35:170
1440299129,2cyclictest14724-21grep22:30:240
1440299129,2cyclictest12718-21sed20:10:260
1440299129,2cyclictest10742-21grep00:05:000
1440299128,3cyclictest6172-21df22:45:160
1440299122,7cyclictest537-21snmpd22:37:160
1440299120,1cyclictest0-21swapper/022:55:120
1441299118,2cyclictest7487-21ssh21:10:033
1441299118,2cyclictest27996-21ssh00:15:023
1441299118,2cyclictest25009-21grep19:30:163
1441299118,2cyclictest22911-21ssh00:30:263
1441299118,2cyclictest18155-21uname19:15:193
1441299118,2cyclictest13456-21tr20:15:143
1441299118,2cyclictest11800-21sed20:10:173
1441299113,4cyclictest18458-21/usr/sbin/munin21:55:163
1441299111,7cyclictest21189-21apt-get22:35:133
1441299110,10cyclictest8170-21rm22:45:523
1441299110,10cyclictest24175-21sh22:17:123
1441099119,1cyclictest27355-21sort19:35:172
1441099119,1cyclictest27115-21ls21:40:202
1441099118,2cyclictest7619-21find20:00:222
1441099118,2cyclictest29955-21cut19:40:162
1441099118,2cyclictest21297-21chrt21:36:572
1441099118,2cyclictest16098-21ssh00:07:412
1441099118,2cyclictest16068-21tr20:20:142
1441099118,2cyclictest11602-21ssh21:11:542
1441099113,3cyclictest225-21systemd-journal20:10:002
1441099110,10cyclictest28317-21sed20:45:132
1441099110,10cyclictest16779-21basename19:15:002
1440599118,2cyclictest9510-21df21:30:151
1440599118,2cyclictest8894-21grep20:05:141
1440599118,2cyclictest31844-21tr19:45:131
1440599118,2cyclictest31016-21ssh23:00:061
1440599118,2cyclictest28934-21munin-run19:40:001
1440599118,2cyclictest27867-21ssh21:40:281
1440599118,2cyclictest22145-21tr19:25:131
1440599118,2cyclictest20890-21wc19:20:221
1440599118,2cyclictest14341-21wc00:25:251
1440599118,2cyclictest11783-21tail20:10:171
1440599118,2cyclictest10164-21cut23:45:141
1440599115,2cyclictest0-21swapper/123:25:131
1440599114,1cyclictest141rcu_preempt21:37:421
1440299119,1cyclictest30544-21grep00:35:160
1440299118,2cyclictest903-21cat23:20:160
1440299118,2cyclictest7374-21ssh22:26:120
1440299118,2cyclictest32491-21ssh00:17:100
1440299118,2cyclictest32406-21uname23:00:200
1440299118,2cyclictest30531-21expr20:50:140
1440299118,2cyclictest30311-21cut19:40:190
1440299118,2cyclictest30290-21grep22:20:240
1440299118,2cyclictest26880-21rm00:33:200
1440299118,2cyclictest24634-21cron23:35:000
1440299118,2cyclictest19219-21chrt23:11:440
1440299118,2cyclictest16758-21uname20:20:190
1440299118,2cyclictest1561-21ssh23:40:140
1440299118,2cyclictest14964-21expr19:10:140
1440299118,2cyclictest14887-21ssh22:11:390
1440299118,2cyclictest14726-21ssh23:47:390
1440299118,2cyclictest10287-21ssh23:06:200
1440299117,3cyclictest3763-21wc21:00:190
1440299116,2cyclictest225-21systemd-journal20:00:000
1441299108,1cyclictest3383-21sort21:00:163
1441299108,1cyclictest273352sleep320:43:413
1441299108,1cyclictest26714-21taskset19:35:053
1441299108,1cyclictest22385-21cat19:25:143
1441299108,1cyclictest10197-21cat20:05:253
1441099108,1cyclictest17284-21cut19:15:142
1441099108,1cyclictest131022sleep220:14:312
1441099107,2cyclictest31545-21taskset19:45:092
1441099107,2cyclictest2929-21tr21:00:142
1441099107,2cyclictest24177-21head20:35:212
1440599108,1cyclictest4415-21sed19:55:161
1440599108,1cyclictest3319-21taskset19:51:011
1440599108,1cyclictest272322sleep120:42:231
1440599108,1cyclictest262722sleep119:32:231
1440599108,1cyclictest22960-21cut20:35:131
1440599107,2cyclictest8141-21chrt20:01:441
1440599107,2cyclictest5886-21hostname21:05:171
1440599100,9cyclictest32226-21taskset20:54:471
1440599100,9cyclictest1671-21which20:55:221
1440299108,1cyclictest24730-21expr19:30:150
1440299108,1cyclictest21292-21grep20:30:170
1440299108,1cyclictest15366-21ssh00:25:580
1440299108,1cyclictest15080-21grep20:15:270
1440299107,2cyclictest26636-21ssh22:00:020
1440299107,2cyclictest26636-21ssh22:00:020
1440299107,2cyclictest26363-21ssh23:35:200
1440299107,2cyclictest24612-21sed20:35:250
144129996,2cyclictest21373-21hostname20:30:183
144029998,1cyclictest25734-21cat20:40:140
144029997,1cyclictest6002-21irqstats21:05:180
144029997,1cyclictest20003-21ls19:20:140
144029997,1cyclictest19192-21chrt19:18:110
144029996,2cyclictest22056-21hostname19:25:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional