You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-09 - 23:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Thu Apr 09, 2026 12:43:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1453429242,45sleep30-21swapper/307:07:373
1450929077,9sleep00-21swapper/007:07:180
1469728370,9sleep20-21swapper/207:09:452
1438228265,13sleep10-21swapper/107:05:381
96492780,0sleep30-21swapper/309:10:223
271072720,0sleep10-21swapper/112:32:401
43712420,0sleep00-21swapper/010:24:440
1480199320,0cyclictest0-21swapper/110:22:541
1480699310,1cyclictest0-21swapper/207:28:292
1480199290,28cyclictest0-21swapper/112:38:161
1480199270,2cyclictest0-21swapper/110:47:201
1480199270,1cyclictest0-21swapper/110:16:381
14806992623,2cyclictest0-21swapper/210:25:122
14806992622,2cyclictest0-21swapper/209:10:222
1480199260,0cyclictest0-21swapper/110:58:461
1480699255,2cyclictest141rcu_preempt09:50:132
1480199250,3cyclictest0-21swapper/112:08:061
1480199250,3cyclictest0-21swapper/111:01:451
1480199250,2cyclictest0-21swapper/109:13:451
1480199250,1cyclictest0-21swapper/110:54:471
1480199233,1cyclictest38150irq/131-snd_hda11:55:221
1480199231,21cyclictest7809-21diskmemload09:22:041
1480199230,23cyclictest0-21swapper/111:18:031
1480199230,1cyclictest0-21swapper/112:27:521
14801992221,0cyclictest0-21swapper/110:00:071
1480199220,21cyclictest0-21swapper/109:35:131
1480199220,0cyclictest0-21swapper/111:28:021
14801992115,5cyclictest7809-21diskmemload09:55:261
1479799218,8cyclictest10423-21munin-run09:50:000
1480199201,17cyclictest7809-21diskmemload09:46:301
1480199191,1cyclictest0-21swapper/109:32:441
1480199190,1cyclictest0-21swapper/110:34:491
230492180,0sleep00-21swapper/011:13:570
14806991810,7cyclictest13352-21perf08:15:002
1480699180,1cyclictest0-21swapper/210:55:572
1480199181,1cyclictest141rcu_preempt12:10:531
1480199180,3cyclictest0-21swapper/111:42:131
1480199180,1cyclictest0-21swapper/111:52:161
1479799189,4cyclictest141rcu_preempt10:03:220
14806991715,1cyclictest4103-21kworker/u8:209:30:102
14806991714,2cyclictest4344-21ssh10:24:432
1480199179,2cyclictest30465-21wc12:15:211
1480199178,3cyclictest18106-21cron08:25:001
1480199170,2cyclictest0-21swapper/111:34:481
1480199170,1cyclictest0-21swapper/111:11:221
1479799170,1cyclictest0-21swapper/012:04:330
14808991614,1cyclictest7809-21diskmemload10:45:143
1480699161,14cyclictest0-21swapper/209:24:522
1480199164,7cyclictest10720-21/usr/sbin/munin11:45:171
1480199163,1cyclictest10838-21apache_accesses09:50:091
1480199161,14cyclictest7809-21diskmemload10:06:541
1480199161,14cyclictest7809-21diskmemload09:27:181
1480199160,16cyclictest0-21swapper/111:06:461
1480199160,14cyclictest0-21swapper/110:38:021
1479799164,6cyclictest25963-21munin-run09:40:000
1479799164,6cyclictest1976-21awk09:45:000
1479799160,11cyclictest0-21swapper/011:25:530
14806991514,1cyclictest0-21swapper/210:04:392
14806991511,4cyclictest32-21ksoftirqd/209:35:532
14806991511,3cyclictest22262-21grep10:35:142
14806991511,2cyclictest19268-21wc11:30:212
1480699150,1cyclictest0-21swapper/211:55:342
1480699150,15cyclictest0-21swapper/212:00:232
1480699150,15cyclictest0-21swapper/211:06:362
1480699150,15cyclictest0-21swapper/210:48:182
1480199156,5cyclictest225-21systemd-journal11:40:001
1480199150,13cyclictest0-21swapper/107:30:131
1479799153,9cyclictest13741-21sensors12:05:220
1479799152,13cyclictest30746-21sh11:18:210
14808991410,3cyclictest27900-21latency_hist08:45:003
14808991410,2cyclictest23597-21tr08:35:143
14808991410,1cyclictest25988-21sendmail-msp09:40:003
1480899140,1cyclictest0-21swapper/310:55:573
1480899140,1cyclictest0-21swapper/310:53:433
1480899140,14cyclictest0-21swapper/312:13:343
1480899140,14cyclictest0-21swapper/309:52:323
1480699148,5cyclictest0-21swapper/212:10:382
1480699141,10cyclictest0-21swapper/209:48:282
14806991410,3cyclictest26752-21sleep07:35:002
14806991410,1cyclictest0-21swapper/211:54:212
1480699140,2cyclictest0-21swapper/211:35:152
1480699140,14cyclictest0-21swapper/212:07:492
1480699140,14cyclictest0-21swapper/209:25:472
1480199149,3cyclictest20747-21tr07:20:171
1480199142,4cyclictest0-21swapper/110:25:131
14801991410,3cyclictest5983-21tr09:05:161
1480199140,1cyclictest0-21swapper/110:10:131
1479799141,9cyclictest0-21swapper/009:10:140
14797991410,3cyclictest32413-21ssh11:00:160
1479799140,1cyclictest0-21swapper/010:40:520
1480899139,3cyclictest16782-21sed08:20:163
1480899138,3cyclictest26680-21tr11:35:153
1480899138,3cyclictest1830-21cut09:25:163
1480899132,6cyclictest0-21swapper/312:25:143
14808991310,2cyclictest30198-21cat10:20:173
14808991310,2cyclictest19409-21cat08:25:203
14806991311,2cyclictest0-21swapper/211:01:532
14806991310,2cyclictest3816-21cron07:55:002
14806991310,2cyclictest3198-21grep11:40:182
14806991310,2cyclictest13090-21sed10:10:162
1480699130,13cyclictest0-21swapper/211:15:242
1480199130,12cyclictest0-21swapper/108:45:131
1480199130,0cyclictest0-21swapper/109:19:091
1479799139,2cyclictest25498-21cron08:40:000
1479799135,6cyclictest11747-21ssh10:10:000
1479799135,4cyclictest0-21swapper/010:10:200
1479799133,9cyclictest15304-21sensors12:25:250
14797991310,2cyclictest526-21cron09:25:000
14797991310,2cyclictest3272-21cut11:40:180
14797991310,2cyclictest24053-21tail10:35:260
14797991310,2cyclictest16142-21grep08:20:130
14797991310,2cyclictest13761-21grep10:30:120
14797991310,2cyclictest12328-21grep08:10:190
14797991310,2cyclictest0-21swapper/010:54:060
1479799130,11cyclictest0-21swapper/010:55:570
1480899129,2cyclictest8897-21ssh11:25:003
1480899129,2cyclictest8623-21wc08:05:003
1480899129,2cyclictest4391-21df10:05:153
1480899129,2cyclictest32581-21uniq11:00:183
1480899129,2cyclictest23008-21ssh09:18:463
1480899129,2cyclictest21203-21grep09:55:253
1480899129,2cyclictest20514-21grep07:20:153
1480899129,2cyclictest20058-21cron12:10:013
1480899129,2cyclictest19053-21tail07:15:263
1480899129,2cyclictest15970-21rm11:09:593
1480899129,2cyclictest14313-21tune2fs10:30:153
1480899129,2cyclictest11222-21ssh09:30:243
1480899128,3cyclictest8340-21ssh10:26:163
1480899128,2cyclictest5351-21cut12:20:133
1480899128,2cyclictest24491-21ls11:15:003
1480899127,3cyclictest25405-21head07:30:163
1480899125,6cyclictest10429-21basename09:49:593
1480899122,8cyclictest28700-21ls10:20:003
1480899120,2cyclictest0-21swapper/311:28:263
1480899120,11cyclictest27091-21sed09:20:273
1480699129,2cyclictest6763-21tr08:00:132
1480699129,2cyclictest30161-21head07:40:182
1480699129,2cyclictest30085-21grep08:45:252
1480699129,2cyclictest27591-21grep07:35:152
1480699129,2cyclictest22946-21head12:30:172
1480699128,2cyclictest10967-21munin-plugin-st08:10:002
1480699127,1cyclictest7809-21diskmemload10:53:432
1480699122,7cyclictest23935-21apache212:38:522
1480699121,2cyclictest27907-21apt-get08:44:592
1480699121,11cyclictest55750irq/128-enp2s0-11:45:022
1480699120,1cyclictest0-21swapper/210:43:152
1480199129,2cyclictest9140-21tr08:05:151
1480199129,2cyclictest7211-21grep08:00:171
1480199129,2cyclictest29672-21tr07:40:151
1480199129,2cyclictest28692-21grep07:35:241
1480199129,2cyclictest26538-21grep08:40:161
1480199129,2cyclictest12452-21tr08:10:221
1480199125,6cyclictest25-21ksoftirqd/107:50:121
14801991210,1cyclictest0-21swapper/109:40:121
1479799129,2cyclictest6393-21wc10:25:190
1479799129,2cyclictest5741-21tail07:55:330
1479799129,2cyclictest32431-21cat12:35:250
1479799129,2cyclictest30443-21head12:15:210
1479799129,2cyclictest22646-21grep08:30:240
1479799129,2cyclictest2052-21wc11:40:000
1479799129,2cyclictest17140-21sh07:15:000
1479799128,3cyclictest7263-21grep08:00:170
1479799128,3cyclictest2772-21wc07:50:190
1479799128,2cyclictest8157-21tr11:05:150
1479799125,3cyclictest225-21systemd-journal12:14:560
1479799124,4cyclictest225-21systemd-journal11:25:000
1479799123,7cyclictest7809-21diskmemload09:30:120
1479799121,3cyclictest141rcu_preempt09:52:040
1480899118,2cyclictest4738-21rm12:00:173
1480899118,2cyclictest3458-21expr09:00:133
1480899118,2cyclictest32754-21ssh12:35:323
1480899118,2cyclictest32542-21ssh10:40:263
1480899118,2cyclictest3232-21head11:40:183
1480899118,2cyclictest31778-21ssh11:57:023
1480899118,2cyclictest28125-21grep10:00:143
1480899118,2cyclictest27670-21sort07:35:153
1480899118,2cyclictest27184-21tune2fs09:40:153
1480899118,2cyclictest24721-21ssh11:15:013
1480899118,2cyclictest24070-21grep10:35:263
1480899118,2cyclictest22961-21grep12:30:173
1480899118,2cyclictest22825-21cut07:25:143
1480899118,2cyclictest20510-21ntpq11:50:203
1480899118,2cyclictest18443-21ssh10:13:403
1480899118,2cyclictest17598-21ssh11:30:013
1480899118,2cyclictest15872-21cat07:10:183
1480899118,2cyclictest15714-21cron08:20:003
1480899118,2cyclictest1401-21tune2fs08:55:173
1480899118,2cyclictest11638-21head11:45:163
1480899117,3cyclictest3135-21sort07:50:243
1480699119,1cyclictest14874-21grep10:30:182
1480699118,2cyclictest9791-21ssh12:22:242
1480699118,2cyclictest6475-21grep09:05:192
1480699118,2cyclictest6225-21readlink08:00:002
1480699118,2cyclictest5352-21grep10:05:222
1480699118,2cyclictest5011-21ssh11:21:592
1480699118,2cyclictest32342-21ssh12:16:272
1480699118,2cyclictest30044-21ssh09:41:392
1480699118,2cyclictest25390-21chrt08:38:392
1480699118,2cyclictest23449-21ssh09:57:032
1480699118,2cyclictest18703-21ssh11:10:362
1480699118,2cyclictest18302-21ssh09:15:222
1480699118,2cyclictest17703-21sed08:20:252
1480699118,2cyclictest16923-21sh12:26:232
1480699118,2cyclictest13866-21grep08:15:142
1480699118,2cyclictest1200-21sed08:55:162
1480699118,2cyclictest11320-21tail11:25:262
1480699112,4cyclictest7809-21diskmemload10:15:232
1480199119,1cyclictest23585-21tr08:35:141
1480199118,2cyclictest6448-21grep12:20:191
1480199118,2cyclictest22918-21grep07:25:151
1480199118,2cyclictest22655-21head08:30:251
1480199118,2cyclictest2246-21wc08:55:261
1480199118,2cyclictest1651-21ssh10:41:341
1480199118,2cyclictest16034-21cat07:10:191
1480199118,2cyclictest1531-21date07:50:001
1480199117,1cyclictest13863-21cut08:15:131
1480199110,3cyclictest0-21swapper/111:23:261
1479799118,2cyclictest9648-21ssh10:46:140
1479799118,2cyclictest5106-21ssh12:20:070
1479799118,2cyclictest4816-21grep09:00:400
1479799118,2cyclictest404-21cut07:45:200
1479799118,2cyclictest31498-21grep08:50:170
1479799118,2cyclictest30481-21cron08:50:000
1479799118,2cyclictest30131-21cat12:35:000
1479799118,2cyclictest30086-21rm11:55:430
1479799118,2cyclictest27179-21grep07:35:130
1479799118,2cyclictest26675-21taskset07:34:290
1479799118,2cyclictest26633-21ssh09:59:300
1479799118,2cyclictest26176-21ssh11:54:130
1479799118,2cyclictest23530-21grep07:25:200
1479799118,2cyclictest21950-21ssh09:17:560
1479799118,2cyclictest21488-21grep10:15:160
1479799118,2cyclictest20079-21grep07:20:130
1479799118,2cyclictest19527-21cat08:25:210
1479799118,2cyclictest18914-21ls11:50:010
1479799118,2cyclictest18272-21expr11:30:140
1479799118,2cyclictest14148-21df08:15:150
1479799117,3cyclictest2899-21wc09:25:250
1480899108,1cyclictest5665-21cat09:05:133
1480899108,1cyclictest32569-21cat07:45:183
1480899108,1cyclictest30748-21mailstats07:40:243
1480899108,1cyclictest29354-21cat08:45:183
1480899108,1cyclictest108982sleep308:09:193
1480899107,2cyclictest4280-21ls07:55:143
1480899107,2cyclictest28817-21ssh12:15:043
1480899107,2cyclictest23059-21taskset08:34:563
1480699108,1cyclictest31193-21ls08:50:152
1480699108,1cyclictest229452sleep208:33:302
1480699108,1cyclictest19843-21wget07:20:112
1480699108,1cyclictest183402sleep208:25:052
1480699108,1cyclictest17416-21apache_volume07:15:122
1480699108,1cyclictest15125-21grep07:10:142
1480699107,2cyclictest5205-21cron09:05:002
1480699107,2cyclictest1276-21taskset07:47:552
1480199108,1cyclictest4102-21cut07:55:131
1480199108,1cyclictest3440-21expr09:00:131
1480199108,1cyclictest30897-21grep08:50:141
1480199107,2cyclictest4364-21grep12:00:131
1480199107,2cyclictest18481-21cut08:25:151
1480199102,2cyclictest452-21systemd-logind07:15:221
1479799108,1cyclictest6335-21tune2fs09:05:180
1479799108,1cyclictest29490-21wget07:40:150
1479799108,1cyclictest17538-21aten2_r7power_v07:15:140
1479799108,1cyclictest10849-21taskset08:08:400
1479799100,9cyclictest1729-21cut08:55:210
148089996,2cyclictest11895-21grep08:10:163
147979997,1cyclictest25759-21chrt08:40:090
148089986,1cyclictest3002sleep308:53:593
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional