You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-07 - 19:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Fri Aug 07, 2026 12:45:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
27321319910096,2cyclictest0-21swapper/007:10:020
2732131999124,37cyclictest2789034-21sh10:57:230
2732131998817,60cyclictest16850irq/134-800000010:07:080
273213199842,75cyclictest2762830-21ssh10:01:470
2732131998319,59cyclictest16850irq/134-800000012:26:550
2732131998319,59cyclictest16850irq/134-800000012:26:550
2732131998318,60cyclictest16850irq/134-800000011:33:110
2732131998038,38cyclictest16850irq/134-800000010:00:020
2732131998028,33cyclictest2806789-21ssh11:35:060
2732131998026,50cyclictest2768307-21ssh10:13:560
2732131998020,36cyclictest0-21swapper/011:24:370
2732131998018,33cyclictest2787838-21date10:55:010
2732131998018,33cyclictest2787838-21date10:55:010
273213199801,49cyclictest2815081-21ssh11:53:010
2732131997832,28cyclictest2782076-21ssh10:42:430
2732131997721,52cyclictest2753363-21ssh09:41:390
273213199758,56cyclictest16850irq/134-800000009:29:430
2732131997530,40cyclictest2826914-21ssh12:17:530
273213199751,69cyclictest2780519-21ssh10:39:430
273213199751,39cyclictest2823082-21munin-run12:10:010
273213199751,2cyclictest0-21swapper/009:50:130
273213199743,34cyclictest2818377-21sed12:00:010
273213199742,67cyclictest3467602-21systemd-journal08:47:110
273213199741,11cyclictest2769073-21modprobe10:15:020
2732131997331,25cyclictest2823676-21ssh12:10:570
273213199732,66cyclictest2776911-21ssh10:31:400
273213199732,34cyclictest2734415-21modprobe07:55:000
2732131997323,33cyclictest2796730-21ssh11:14:040
2732131997312,53cyclictest16850irq/134-800000009:20:020
2732140997223,44cyclictest275-21dbus-daemon11:45:012
2732136997229,7cyclictest2787765-21ssh10:55:011
2732136997229,7cyclictest2787765-21ssh10:55:001
2732131997230,38cyclictest2830086-21ssh12:24:580
273213199722,53cyclictest2819939-21ssh12:03:170
2732131997222,45cyclictest16850irq/134-800000009:47:350
2732131997221,47cyclictest16850irq/134-800000009:22:110
273213199721,68cyclictest0-21swapper/011:15:030
2732131997129,25cyclictest2813314-21ssh11:49:160
2732131997127,28cyclictest2752016-21ssh09:39:110
2732131997114,52cyclictest2772605-21ssh10:22:400
2732131997032,34cyclictest2774039-21ssh10:25:310
273213199701,5cyclictest2792545-21munin-run11:05:010
273213199701,5cyclictest2792545-21munin-run11:05:010
273213199701,12cyclictest2738428-21latency_hist09:10:020
273213199701,12cyclictest2738428-21latency_hist09:10:020
2732140996927,38cyclictest2826501-21sh12:17:032
2732131996829,35cyclictest0-21swapper/012:32:130
2732131996819,44cyclictest16850irq/134-800000009:34:510
273213199662,59cyclictest2732316-21cron07:15:000
2732131996620,42cyclictest16850irq/134-800000011:44:190
273213199661,4cyclictest0-21swapper/009:10:260
2732131996614,46cyclictest16850irq/134-800000011:25:100
2732131996523,36cyclictest2733665-21systemd-run07:40:010
273213199651,6cyclictest2792819-21ssh11:05:160
2732131996515,30cyclictest19350irq/34-mmc107:50:010
273213199647,51cyclictest16850irq/134-800000010:47:460
273213199642,57cyclictest2736579-21latency_hist08:35:020
2732131996424,35cyclictest2736225-21sshd08:29:120
2732140996130,26cyclictest3467602-21systemd-journal07:56:112
2732131996120,37cyclictest2735765-21latency_hist08:20:020
2732140996027,29cyclictest2831806-21sh12:28:272
2732140996027,29cyclictest2831806-21sh12:28:272
2732131996033,23cyclictest1-21systemd08:55:000
273213199602,53cyclictest2735440-21cron08:15:000
273213199601,3cyclictest0-21swapper/012:35:320
2732140995928,9cyclictest2771662-21kworker/2:110:50:002
2732140995922,33cyclictest2738322-21diskmemload11:28:552
2732131995929,25cyclictest275-21dbus-daemon07:20:010
2732131995929,25cyclictest275-21dbus-daemon07:20:010
2732131995912,41cyclictest1-21systemd07:25:000
273214099583,7cyclictest1-21systemd12:10:012
273214099582,19cyclictest2801882-21munin-run11:25:012
2732140995820,34cyclictest2759938-21sh09:55:322
2732140995815,38cyclictest2763914-1kworker/u9:110:19:272
273213199581,2cyclictest0-21swapper/007:31:090
273214099577,43cyclictest275-21dbus-daemon10:35:012
273214099572,16cyclictest2834790-21ssh12:34:532
273214099572,13cyclictest2817173-21ssh11:57:182
2732140995719,34cyclictest2738322-21diskmemload12:03:042
2732131995727,26cyclictest2737811-21grep09:00:000
2732131995727,26cyclictest2737811-21grep09:00:000
273213199571,51cyclictest2738067-21munin-run09:05:000
273214099564,10cyclictest30-21ksoftirqd/210:06:242
2732140995622,30cyclictest30-21ksoftirqd/210:42:322
273214099562,17cyclictest2830060-21ssh12:24:542
2732140995619,33cyclictest2741236-21sh09:15:592
273214099561,7cyclictest2795169-21ssh11:10:162
2732140995614,37cyclictest2753328-1kworker/u9:010:24:102
2732131995638,13cyclictest12-21ksoftirqd/008:40:280
273213199562,5cyclictest2736775-21lockfile-create08:40:000
273213199561,13cyclictest2733138-21munin-run07:30:010
273214099553,7cyclictest2764156-21modprobe10:05:002
273214099553,16cyclictest2837193-21ssh12:40:012
273214099553,15cyclictest284-21systemd-logind09:10:012
273214099553,15cyclictest284-21systemd-logind09:10:012
2732140995527,23cyclictest2824991-21sh12:14:062
273214099552,20cyclictest2767823-21ssh10:12:432
273214099552,17cyclictest2813226-21ssh11:49:052
273214099552,17cyclictest2756686-21ssh09:49:022
273214099552,14cyclictest2738593-21ssh09:10:062
2732140995516,35cyclictest30-21ksoftirqd/209:05:002
273214099551,17cyclictest2787720-21ssh10:54:532
273214099551,17cyclictest2787720-21ssh10:54:532
273214099551,14cyclictest2816107-21ssh11:55:032
273213199552,48cyclictest2734962-21/usr/sbin/munin08:05:010
273214499546,43cyclictest1-21systemd10:15:013
273214099542,8cyclictest2732561-21sshd07:17:102
273214099542,8cyclictest2732561-21sshd07:17:102
273214099542,5cyclictest3467602-21systemd-journal08:20:022
273214099542,16cyclictest2793124-21ssh11:05:562
273214099541,20cyclictest2808592-21ssh11:39:252
273214099541,15cyclictest2759290-21ssh09:54:292
273214099541,14cyclictest2798120-21sh11:16:412
273213199541,48cyclictest2735961-21sshd08:21:110
2732144995337,12cyclictest1-21systemd11:10:003
273214099533,8cyclictest3467602-21systemd-journal07:45:012
273214099533,4cyclictest2734932-21sshd08:04:242
273214099532,16cyclictest2736754-21cat08:40:002
273214099531,9cyclictest2753277-21ssh09:41:282
2732140995319,30cyclictest2788477-21sshd10:56:132
2732140995312,37cyclictest2750324-21sh09:35:092
273214099531,13cyclictest2775989-21munin-run10:30:012
2732136995337,12cyclictest2837159-21run-parts12:40:011
2732136995334,15cyclictest1-21systemd07:30:011
273214099525,40cyclictest0-21swapper/207:40:002
273214099522,6cyclictest3467602-21systemd-journal07:30:012
273214099522,14cyclictest2746773-21ssh09:27:472
273214099522,13cyclictest2779752-21sh10:37:562
273214099522,12cyclictest2791283-21ssh11:02:062
273214099522,12cyclictest2791283-21ssh11:02:062
2732140995212,34cyclictest3983913-1kworker/2:2H11:30:112
2732136995234,13cyclictest3467602-21systemd-journal07:25:001
2732136995234,13cyclictest275-21dbus-daemon09:30:011
273213199522,45cyclictest2735200-21munin-run08:10:000
2732144995134,13cyclictest2780690-21systemctl10:40:013
2732144995132,14cyclictest2837139-21cron12:40:003
2732144995130,15cyclictest3467602-21systemd-journal11:55:003
2732144995128,16cyclictest275-21dbus-daemon12:05:023
273214099512,4cyclictest364-21snmpd08:28:352
273214099512,12cyclictest2735606-21sed08:15:022
273214099511,12cyclictest2748976-21ssh09:32:272
273214099501,14cyclictest2745031-21ssh09:24:122
273214099501,12cyclictest2733377-21latency_hist07:35:002
2732136995031,14cyclictest275-21dbus-daemon08:25:011
2732131995023,24cyclictest1-21systemd07:42:100
273213199501,5cyclictest2734646-21sshd07:58:110
2732144994941,5cyclictest37-21ksoftirqd/310:06:113
2732140994922,23cyclictest2736183-21latency_hist08:25:022
273214099492,14cyclictest2737545-21munin-run08:55:012
273214099492,14cyclictest2735382-21latency_hist08:10:022
273214099492,12cyclictest1-21systemd09:00:012
273214099492,12cyclictest1-21systemd09:00:012
273214099491,5cyclictest2736485-21cron08:35:012
2732136994939,6cyclictest2738322-21diskmemload12:30:011
2732136994939,6cyclictest2738322-21diskmemload12:30:001
2732136994931,13cyclictest1-21systemd10:10:011
2732144994832,11cyclictest2804202-21diskmemload11:30:003
2732144994831,13cyclictest1-21systemd12:20:013
2732140994821,23cyclictest309-21in:imuxsock07:50:002
273214099482,11cyclictest2732848-21modprobe07:23:102
273214099481,15cyclictest2732304-21sshd07:13:102
2732144994717,16cyclictest2732125-21cyclictest08:15:023
273214099473,6cyclictest275-21dbus-daemon08:50:002
273214099472,5cyclictest0-21swapper/208:40:122
2732136994634,6cyclictest309-21in:imuxsock08:40:001
2732136994631,11cyclictest2761818-21readlink10:00:001
2732136994631,10cyclictest275-21dbus-daemon07:20:011
2732136994631,10cyclictest275-21dbus-daemon07:20:011
273213699462,4cyclictest131rcu_preempt11:55:021
273214499456,11cyclictest37-21ksoftirqd/309:40:013
2732144994532,10cyclictest37-21ksoftirqd/311:35:013
2732144994520,17cyclictest275-21dbus-daemon07:50:003
273214099452,7cyclictest2734422-21systemd07:55:002
2732140994520,23cyclictest30-21ksoftirqd/207:10:012
2732136994534,7cyclictest23-21ksoftirqd/111:43:071
2732136994531,11cyclictest23-21ksoftirqd/110:03:551
2732136994529,11cyclictest1-21systemd07:45:011
2732136994527,14cyclictest1-21systemd09:25:011
273214499443,14cyclictest37-21ksoftirqd/312:33:423
2732144994429,11cyclictest3467602-21systemd-journal09:15:113
2732144994421,16cyclictest275-21dbus-daemon10:00:023
2732136994427,11cyclictest275-21dbus-daemon12:15:011
273213699441,38cyclictest23-21ksoftirqd/110:27:111
2732144994331,8cyclictest37-21ksoftirqd/309:29:573
2732144994329,9cyclictest309-21in:imuxsock12:10:033
2732144994328,11cyclictest1-21systemd07:20:013
2732144994328,11cyclictest1-21systemd07:20:013
2732144994327,11cyclictest2790117-21modprobe11:00:003
2732144994326,12cyclictest1-21systemd11:05:013
2732144994326,12cyclictest1-21systemd11:05:013
2732144994320,14cyclictest3467602-21systemd-journal11:40:003
2732144994314,6cyclictest131rcu_preempt09:25:013
273214499431,39cyclictest1-21systemd09:55:003
2732136994332,7cyclictest2803046-21ssh11:27:121
2732136994324,13cyclictest1-21systemd11:15:011
273214499423,6cyclictest131rcu_preempt11:50:023
273214499423,6cyclictest131rcu_preempt08:35:023
2732144994233,6cyclictest37-21ksoftirqd/311:16:473
2732144994230,8cyclictest37-21ksoftirqd/309:15:003
2732144994227,12cyclictest37-21ksoftirqd/310:17:483
2732136994232,6cyclictest274-21cron09:10:001
2732136994232,6cyclictest274-21cron09:10:001
2732136994224,14cyclictest275-21dbus-daemon10:20:001
2732136994215,5cyclictest131rcu_preempt09:11:381
2732136994212,6cyclictest131rcu_preempt10:24:061
273213699421,24cyclictest0-21swapper/112:15:031
273214499414,7cyclictest131rcu_preempt11:43:483
2732144994125,11cyclictest37-21ksoftirqd/312:22:253
2732144994124,11cyclictest3467602-21systemd-journal08:25:003
2732144994121,14cyclictest3467602-21systemd-journal07:30:013
2732144994120,15cyclictest275-21dbus-daemon09:35:013
273213699414,6cyclictest131rcu_preempt09:40:011
273213699413,4cyclictest131rcu_preempt10:38:351
2732136994133,6cyclictest23-21ksoftirqd/112:10:011
2732136994130,8cyclictest23-21ksoftirqd/112:05:011
2732136994127,11cyclictest23-21ksoftirqd/111:32:071
2732136994123,11cyclictest0-21swapper/111:10:011
2732144994029,8cyclictest37-21ksoftirqd/311:13:403
2732144994029,7cyclictest37-21ksoftirqd/312:00:023
273214499401,5cyclictest131rcu_preempt10:45:003
2732136994030,7cyclictest2813785-21cat11:50:031
2732136994030,7cyclictest23-21ksoftirqd/108:10:011
2732136994029,8cyclictest2747985-21sshd09:30:111
2732136994026,9cyclictest309-21in:imuxsock10:56:121
2732136994026,10cyclictest23-21ksoftirqd/112:25:021
2732136994026,10cyclictest23-21ksoftirqd/109:46:151
273214499394,7cyclictest37-21ksoftirqd/308:00:003
2732144993927,9cyclictest37-21ksoftirqd/309:47:003
2732144993926,9cyclictest37-21ksoftirqd/310:26:213
2732144993925,11cyclictest37-21ksoftirqd/310:03:453
273214499391,3cyclictest131rcu_preempt10:21:213
2732144993912,5cyclictest131rcu_preempt12:25:043
2732144993912,5cyclictest131rcu_preempt12:25:043
2732136993931,5cyclictest2733375-21latency_hist07:35:001
2732136993929,8cyclictest23-21ksoftirqd/111:17:061
2732136993929,7cyclictest2785409-21sh10:50:011
2732136993927,9cyclictest23-21ksoftirqd/110:15:001
2732136993925,10cyclictest23-21ksoftirqd/110:31:191
2732136993924,11cyclictest23-21ksoftirqd/112:32:541
2732144993829,5cyclictest2737022-21sh08:45:003
2732144993827,8cyclictest37-21ksoftirqd/308:05:113
2732144993824,11cyclictest37-21ksoftirqd/312:05:383
2732144993823,11cyclictest37-21ksoftirqd/309:42:473
273214499381,4cyclictest131rcu_preempt11:23:533
2732136993827,7cyclictest2734410-21cron07:55:001
2732136993826,10cyclictest23-21ksoftirqd/111:04:301
2732136993826,10cyclictest23-21ksoftirqd/111:04:301
2732136993825,9cyclictest23-21ksoftirqd/108:01:111
2732136993824,11cyclictest23-21ksoftirqd/110:44:211
2732136993824,10cyclictest23-21ksoftirqd/107:40:001
273213699381,14cyclictest2740975-21ssh09:15:141
273214499379,6cyclictest131rcu_preempt10:47:443
273214499378,16cyclictest2738315-21modprobe09:10:003
273214499378,16cyclictest2738315-21modprobe09:10:003
2732144993717,3cyclictest37-21ksoftirqd/310:31:293
2732136993727,8cyclictest2732291-21sshd07:10:101
273213699372,5cyclictest131rcu_preempt07:10:011
2732136993725,8cyclictest2816315-21ssh11:55:181
2732136993724,10cyclictest23-21ksoftirqd/111:21:311
2732136993721,13cyclictest23-21ksoftirqd/109:45:011
273213699371,12cyclictest280703550ssh11:35:491
273214499367,25cyclictest2738079-21unin-run09:05:003
273214499365,7cyclictest131rcu_preempt07:55:003
2732144993615,15cyclictest275-21dbus-daemon09:00:003
2732144993615,15cyclictest275-21dbus-daemon09:00:003
2732136993628,5cyclictest2757532-21sh09:50:271
2732136993625,9cyclictest23-21ksoftirqd/108:20:031
2732136993625,8cyclictest23-21ksoftirqd/109:00:021
2732136993625,8cyclictest23-21ksoftirqd/109:00:021
2732136993623,9cyclictest3467602-21systemd-journal08:49:331
2732136993623,9cyclictest23-21ksoftirqd/108:32:111
2732136993619,13cyclictest1-21systemd09:05:001
2732136993610,17cyclictest0-21swapper/108:00:001
2732144993521,9cyclictest309-21in:imuxsock08:46:113
2732144993520,9cyclictest0-21swapper/308:05:013
2732144993520,11cyclictest2737687-21sed08:55:023
2732136993524,8cyclictest23-21ksoftirqd/108:55:011
2732136993517,13cyclictest275-21dbus-daemon08:15:011
2732144993423,8cyclictest37-21ksoftirqd/310:52:593
2732144993423,8cyclictest37-21ksoftirqd/310:52:593
2732136993423,9cyclictest23-21ksoftirqd/108:30:021
2732136993417,13cyclictest23-21ksoftirqd/108:45:001
2732144993324,5cyclictest2732456-21latency_hist07:15:013
2732136993325,5cyclictest2734295-21latency_hist07:50:021
273214499324,6cyclictest131rcu_preempt08:30:023
273214499323,5cyclictest131rcu_preempt07:40:003
2732144993225,4cyclictest2734047-21latency_hist07:45:023
2732144993217,12cyclictest1-21systemd08:40:013
273214499314,5cyclictest131rcu_preempt08:20:023
2732144993124,5cyclictest2732164-21latency_hist07:10:013
2732144993123,6cyclictest37-21ksoftirqd/307:25:003
2732144992814,10cyclictest2733370-21modprobe07:35:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional