You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-09 - 21:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Mon Mar 09, 2026 12:45:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
978423991151,111cyclictest0-21swapper/011:00:020
978423999894,2cyclictest0-21swapper/009:15:020
97843399932,87cyclictest3467602-21systemd-journal11:45:032
978433998624,46cyclictest0-21swapper/209:50:022
978433997923,17cyclictest990458-21ssh09:22:412
978433997922,17cyclictest1024233-21ssh10:38:012
978433997824,21cyclictest1073281-21ssh12:27:352
978433997725,13cyclictest1015335-21ssh10:18:042
978433997532,17cyclictest1032414-21ssh10:55:572
978433997521,15cyclictest1040125-21ssh11:13:202
97842399746,30cyclictest1074474-21latency_hist12:30:010
978433997221,16cyclictest1045573-21ssh11:25:042
978433997221,13cyclictest987981-21ssh09:17:102
978433997126,14cyclictest986173-21ssh09:13:162
978423997145,16cyclictest1076535-21sh12:35:020
978433997022,13cyclictest1066616-21rm12:12:482
978433997022,13cyclictest1066616-21rm12:12:472
978433997021,18cyclictest1018032-21ssh10:24:112
978433997020,16cyclictest1054662-21ssh11:45:542
978433996921,17cyclictest1057724-21rm11:53:022
978433996921,14cyclictest1036806-21ssh11:05:362
978433996918,15cyclictest1031426-21ssh10:53:592
978433996918,15cyclictest1031426-21ssh10:53:592
978433996851,14cyclictest1044253-21ssh11:22:252
978433996828,9cyclictest1072068-21munin-run12:25:012
97842399675,45cyclictest12-21ksoftirqd/011:54:460
978433996642,19cyclictest1026161-21ssh10:42:062
978433996627,12cyclictest997524-21ssh09:38:322
978423996618,11cyclictest12-21ksoftirqd/011:36:520
978433996526,6cyclictest1003099-21ssh09:50:352
978433996524,13cyclictest984601-21diskmemload09:32:292
978433996524,13cyclictest984601-21diskmemload09:32:282
978433996521,16cyclictest1063024-21systemd-run12:05:012
978433996521,16cyclictest1063024-21systemd-run12:05:002
978423996417,10cyclictest12-21ksoftirqd/010:50:360
978423996417,10cyclictest12-21ksoftirqd/010:50:360
978433996348,12cyclictest1007955-21ssh10:01:302
978433996344,15cyclictest992646-21ssh09:27:272
978433996343,16cyclictest1059693-21ssh11:57:222
978433996319,14cyclictest1048118-21ssh11:30:432
978423996320,10cyclictest3467602-21systemd-journal10:47:310
978423996319,9cyclictest12-21ksoftirqd/009:13:420
978433996219,13cyclictest1067803-21ssh12:15:092
97842399629,15cyclictest1023799-21ssh10:37:010
97842399624,54cyclictest1049307-21ssh11:33:440
978423996236,21cyclictest1045627-21ssh11:25:130
978433996149,9cyclictest1078300-21ssh12:39:032
978433996144,13cyclictest1042277-21ssh11:18:042
978433996144,13cyclictest1028522-21ssh10:47:292
978433996143,15cyclictest1035145-21ssh11:02:042
978433996143,14cyclictest1021174-21ssh10:30:502
978433996143,14cyclictest1021174-21ssh10:30:492
978433996143,14cyclictest1019730-21ssh10:27:492
978433996143,14cyclictest1005514-21ssh09:56:022
978433996117,14cyclictest1013357-21ssh10:13:482
978433996117,14cyclictest1013357-21ssh10:13:472
978433996116,12cyclictest984601-21diskmemload12:30:052
978423996114,10cyclictest12-21ksoftirqd/008:35:020
978423996113,10cyclictest12-21ksoftirqd/012:16:500
978433996044,13cyclictest1051444-21ssh11:39:032
978433996042,14cyclictest1063470-21ssh12:05:362
978433996040,16cyclictest998519-21ssh09:40:262
978433996040,16cyclictest998519-21ssh09:40:252
978423996031,13cyclictest989698-21ssh09:20:440
978423996013,10cyclictest12-21ksoftirqd/012:02:500
978423996013,10cyclictest12-21ksoftirqd/012:02:500
978433995943,13cyclictest981215-21sshd08:04:082
978433995943,12cyclictest981962-21cron08:17:002
978433995942,13cyclictest1009636-21ssh10:05:052
978423995928,27cyclictest16850irq/134-800000010:19:550
978423995919,36cyclictest0-21swapper/011:17:160
978423995915,7cyclictest12-21ksoftirqd/011:13:110
978423995912,12cyclictest3467602-21systemd-journal11:45:010
978433995845,10cyclictest979642-21sshd07:32:082
978433995843,11cyclictest980670-21sshd07:50:242
978423995824,6cyclictest12-21ksoftirqd/011:24:530
978423995816,10cyclictest12-21ksoftirqd/009:51:060
978438995735,17cyclictest275-21dbus-daemon07:20:003
978433995741,13cyclictest979888-21sshd07:36:092
978433995740,14cyclictest980414-21sshd07:45:082
978423995736,16cyclictest1-21systemd11:05:000
978423995716,11cyclictest12-21ksoftirqd/011:46:450
978423995715,8cyclictest12-21ksoftirqd/012:08:470
978423995711,42cyclictest1060878-21date12:00:020
978433995642,10cyclictest984110-21munin-run09:00:022
978433995629,21cyclictest979153-21modprobe07:25:012
978423995619,33cyclictest16850irq/134-800000012:30:010
978423995614,9cyclictest12-21ksoftirqd/010:34:180
978423995614,9cyclictest12-21ksoftirqd/010:34:170
978433995542,9cyclictest981450-21sshd08:05:252
978433995542,9cyclictest364-21snmpd08:27:252
978433995542,10cyclictest979392-21sshd07:28:082
978433995538,13cyclictest978866-21sshd07:19:062
978423995527,12cyclictest1025761-21ssh10:41:100
978433995441,9cyclictest984361-21munin-run09:05:012
978433995441,9cyclictest984361-21munin-run09:05:012
978433995439,11cyclictest982745-21sshd08:30:082
978433995438,12cyclictest982266-21cron08:24:592
978423995444,7cyclictest12-21ksoftirqd/009:36:120
978433995341,8cyclictest980212-21munin-run07:45:012
978433995334,15cyclictest980934-21sendmail-mta07:58:502
97842899539,29cyclictest1050548-21kworker/u8:111:40:011
978428995336,13cyclictest1-21systemd12:00:001
97842399536,15cyclictest1-21systemd08:50:010
978423995319,29cyclictest16850irq/134-800000010:02:420
978423995315,9cyclictest12-21ksoftirqd/012:22:250
978438995237,11cyclictest275-21dbus-daemon07:35:013
978438995232,15cyclictest1043060-21cron11:20:003
978433995238,9cyclictest48-21kcompactd008:54:362
978433995233,15cyclictest364-21snmpd07:14:322
978423995245,4cyclictest151rcuc/009:48:560
978423995242,5cyclictest16950irq/224-800000009:09:590
978423995214,7cyclictest12-21ksoftirqd/010:27:080
978423995211,7cyclictest12-21ksoftirqd/012:10:260
978423995211,7cyclictest12-21ksoftirqd/012:10:250
978433995141,6cyclictest983282-21sshd08:41:072
978433995138,9cyclictest984628-21munin-run09:10:022
978428995136,11cyclictest1-21systemd10:00:011
978428995132,14cyclictest3467602-21systemd-journal08:40:011
978423995125,18cyclictest3467602-21systemd-journal07:30:010
978438995031,15cyclictest3467602-21systemd-journal11:15:013
978433995037,10cyclictest364-21snmpd08:46:002
978423995021,5cyclictest1012051-21rm10:10:320
978423995021,5cyclictest1012051-21rm10:10:310
97842399501,16cyclictest979927-21grep07:40:000
978423995011,36cyclictest1006753-21sshd09:59:080
978438994937,8cyclictest3467602-21systemd-journal12:05:013
978438994937,8cyclictest3467602-21systemd-journal12:05:003
978433994931,14cyclictest48-21kcompactd008:10:132
978428994934,11cyclictest1-21systemd09:55:011
978423994940,5cyclictest16950irq/224-800000008:00:000
97842399493,42cyclictest1009950-21rm10:05:480
978423994921,13cyclictest993397-21ssh09:29:240
978438994835,10cyclictest275-21dbus-daemon09:25:023
978438994831,14cyclictest1077576-21sshd12:37:123
978433994824,17cyclictest983052-21sendmail08:40:002
978428994834,10cyclictest23-21ksoftirqd/108:00:001
97843899473,41cyclictest1031844-21ssh10:55:033
97843899473,41cyclictest1031844-21ssh10:55:023
978438994733,10cyclictest1060755-21systemctl11:59:593
978438994731,12cyclictest3467602-21systemd-journal12:25:003
978433994726,18cyclictest978365-21latency_hist07:10:012
978428994733,11cyclictest3467602-21systemd-journal11:40:031
978428994732,12cyclictest23-21ksoftirqd/108:35:001
978423994731,12cyclictest1-21systemd08:45:010
978423994729,13cyclictest3467602-21systemd-journal08:15:000
97842399472,4cyclictest998495-21ssh09:40:230
97842399472,4cyclictest998495-21ssh09:40:220
978423994719,24cyclictest12-21ksoftirqd/007:35:020
978438994631,12cyclictest37-21ksoftirqd/312:07:153
978438994630,12cyclictest37-21ksoftirqd/309:00:003
97842899462,4cyclictest131rcu_preempt11:50:251
97842399469,34cyclictest16850irq/134-800000009:35:030
97842399469,34cyclictest16850irq/134-800000009:35:030
978423994631,11cyclictest981214-21sshd08:03:080
978423994629,14cyclictest1017614-21ssh10:23:110
978423994611,30cyclictest16850irq/134-800000011:06:000
978438994534,9cyclictest37-21ksoftirqd/311:45:013
978438994512,6cyclictest131rcu_preempt09:57:083
97842899454,3cyclictest131rcu_preempt10:04:211
97842899453,3cyclictest131rcu_preempt12:26:081
978428994531,11cyclictest23-21ksoftirqd/109:35:001
978428994531,11cyclictest23-21ksoftirqd/109:35:001
978428994530,11cyclictest1-21systemd07:40:001
978423994538,4cyclictest16950irq/224-800000009:00:000
978423994527,14cyclictest980165-21sshd07:40:220
97843899446,6cyclictest131rcu_preempt08:40:013
97843899444,3cyclictest131rcu_preempt11:26:083
978438994428,13cyclictest3467602-21systemd-journal10:20:003
978438994423,15cyclictest275-21dbus-daemon08:15:003
978428994434,7cyclictest991666-21ssh09:25:091
978428994430,11cyclictest1010704-21ssh10:07:431
97842399443,11cyclictest983014-21sshd08:39:080
978423994431,10cyclictest980677-21sshd07:52:090
978423994414,8cyclictest12-21ksoftirqd/007:19:050
97843899434,3cyclictest131rcu_preempt09:32:453
97843899434,3cyclictest131rcu_preempt09:32:443
97843899434,16cyclictest1051867-21munin-run11:39:593
978438994331,7cyclictest1045491-21latency_hist11:25:023
978438994326,13cyclictest1-21systemd07:55:003
978438994320,5cyclictest37-21ksoftirqd/311:53:193
978428994332,8cyclictest23-21ksoftirqd/112:30:161
97842899431,3cyclictest131rcu_preempt10:26:201
97842899431,3cyclictest131rcu_preempt07:33:091
978423994325,14cyclictest982025-21sendmail-msp08:20:000
978438994228,9cyclictest309-21in:imuxsock10:10:003
978438994226,12cyclictest3467602-21systemd-journal08:10:013
97843899421,4cyclictest131rcu_preempt10:37:053
97843899417,5cyclictest131rcu_preempt12:30:003
978438994125,13cyclictest998151-21diskmemload09:40:013
978438994120,5cyclictest37-21ksoftirqd/310:45:013
978428994126,12cyclictest23-21ksoftirqd/109:45:151
978428994125,13cyclictest23-21ksoftirqd/108:20:001
978428994125,12cyclictest275-21dbus-daemon07:15:011
978428994125,11cyclictest23-21ksoftirqd/108:30:011
97842899412,3cyclictest131rcu_preempt10:50:131
97842899412,3cyclictest131rcu_preempt10:50:121
97842899411,3cyclictest131rcu_preempt11:08:181
978423994125,12cyclictest983787-21sshd08:51:080
978438994029,6cyclictest309-21in:imuxsock10:50:003
978438994028,9cyclictest981986-21cron08:20:003
97843899402,5cyclictest131rcu_preempt09:55:013
978438994025,11cyclictest37-21ksoftirqd/309:10:003
978438994024,10cyclictest275-21dbus-daemon08:30:003
978438994022,13cyclictest3467602-21systemd-journal10:30:003
978428994032,6cyclictest23-21ksoftirqd/110:18:321
978428994028,7cyclictest309-21in:imuxsock08:55:031
978428994026,11cyclictest979388-21sshd07:27:081
978428994025,11cyclictest275-21dbus-daemon10:45:011
978428994024,12cyclictest3467602-21systemd-journal11:35:011
97842899401,4cyclictest131rcu_preempt09:35:221
97842899401,3cyclictest131rcu_preempt11:22:591
97842899401,3cyclictest131rcu_preempt11:10:251
978423994027,9cyclictest982260-21sshd08:24:090
978423994026,11cyclictest979131-21sshd07:20:200
978423994024,12cyclictest982506-21sshd08:28:080
978438993929,7cyclictest37-21ksoftirqd/307:12:363
978438993927,9cyclictest37-21ksoftirqd/311:50:033
978438993927,10cyclictest37-21ksoftirqd/312:19:473
978438993924,13cyclictest37-21ksoftirqd/312:33:393
978438993923,12cyclictest1-21systemd10:15:013
978438993923,12cyclictest1-21systemd10:15:013
978438993910,5cyclictest131rcu_preempt11:09:193
97842899393,3cyclictest131rcu_preempt12:22:121
97842899393,3cyclictest131rcu_preempt10:49:061
978428993928,6cyclictest309-21in:imuxsock11:20:011
978428993923,10cyclictest275-21dbus-daemon10:25:011
978423993913,20cyclictest980652-21latency_hist07:50:020
97843899389,6cyclictest131rcu_preempt09:19:403
97843899386,22cyclictest979913-21modprobe07:40:003
978438993830,6cyclictest37-21ksoftirqd/311:02:253
978438993827,9cyclictest37-21ksoftirqd/312:15:023
978438993827,9cyclictest37-21ksoftirqd/312:15:013
978438993824,11cyclictest37-21ksoftirqd/310:55:023
978438993822,12cyclictest1-21systemd10:05:003
978438993816,4cyclictest37-21ksoftirqd/309:28:193
978438993816,12cyclictest3467718-21lldpd10:32:073
978438993816,12cyclictest3467718-21lldpd10:32:063
978438993811,6cyclictest131rcu_preempt09:14:353
978428993830,5cyclictest1054900-21sh11:46:291
978428993829,6cyclictest989117-21sh09:19:571
978428993828,8cyclictest23-21ksoftirqd/110:57:271
978428993826,9cyclictest23-21ksoftirqd/111:04:171
978428993824,11cyclictest23-21ksoftirqd/112:04:101
978428993824,11cyclictest23-21ksoftirqd/112:04:101
97842899381,3cyclictest131rcu_preempt09:22:211
978428993813,16cyclictest23-21ksoftirqd/112:36:121
978423993830,5cyclictest978625-21munin-run07:15:010
978423993826,9cyclictest984330-21sshd09:04:070
978423993826,9cyclictest984330-21sshd09:04:070
978438993719,4cyclictest37-21ksoftirqd/308:02:463
978428993727,7cyclictest23-21ksoftirqd/108:55:031
978428993724,9cyclictest980186-21cron07:45:001
978428993724,10cyclictest23-21ksoftirqd/108:05:131
978428993723,9cyclictest309-21in:imuxsock08:45:011
978428993722,11cyclictest275-21dbus-daemon09:15:011
97842899371,4cyclictest131rcu_preempt11:26:541
97842899371,3cyclictest131rcu_preempt10:11:561
97842899371,3cyclictest131rcu_preempt10:11:561
978428993713,8cyclictest0-21swapper/112:07:011
97842399372,29cyclictest3467602-21systemd-journal08:05:030
978438993625,9cyclictest37-21ksoftirqd/309:40:193
978438993625,9cyclictest37-21ksoftirqd/309:40:183
97842899364,3cyclictest131rcu_preempt08:15:011
978428993618,13cyclictest275-21dbus-daemon07:25:001
97842899361,7cyclictest0-21swapper/110:34:281
97842899361,7cyclictest0-21swapper/110:34:271
97842899361,3cyclictest131rcu_preempt09:42:091
97842899361,3cyclictest131rcu_preempt09:42:081
97843899355,5cyclictest131rcu_preempt09:46:033
978438993526,6cyclictest1016786-21sh10:21:053
978438993524,8cyclictest0-21swapper/308:00:033
97843899352,29cyclictest275-21dbus-daemon11:35:003
978438993521,9cyclictest3467602-21systemd-journal07:40:233
978428993523,9cyclictest23-21ksoftirqd/109:06:071
978428993522,9cyclictest23-21ksoftirqd/112:16:581
97842899351,3cyclictest131rcu_preempt12:14:261
97842899351,3cyclictest131rcu_preempt12:14:251
978438993425,7cyclictest37-21ksoftirqd/308:50:013
978438993422,7cyclictest309-21in:imuxsock08:25:013
978438993410,6cyclictest131rcu_preempt08:30:073
978428993423,9cyclictest23-21ksoftirqd/110:38:021
978428993422,9cyclictest983537-21sshd08:46:081
978428993422,9cyclictest23-21ksoftirqd/109:00:091
978428993422,9cyclictest23-21ksoftirqd/109:00:081
97842899341,16cyclictest978897-21systemctl07:20:001
978438993326,4cyclictest979342-21latency_hist07:25:023
978438993323,5cyclictest37-21ksoftirqd/307:50:013
978438993322,8cyclictest37-21ksoftirqd/307:10:023
978438993321,9cyclictest37-21ksoftirqd/309:00:053
978438993321,9cyclictest37-21ksoftirqd/309:00:043
978438993319,10cyclictest275-21dbus-daemon07:30:013
978428993323,4cyclictest23-21ksoftirqd/107:55:001
978438993223,5cyclictest37-21ksoftirqd/308:45:013
97842899322,3cyclictest131rcu_preempt08:25:011
978428993221,8cyclictest23-21ksoftirqd/107:49:421
97842399321,28cyclictest0-21swapper/007:10:010
978428993116,12cyclictest3467602-21systemd-journal08:03:081
978438993020,5cyclictest37-21ksoftirqd/308:55:003
97842899284,18cyclictest978432-21latency_hist07:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional