You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-14 - 04:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Thu May 14, 2026 00:45:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
385028999770,24cyclictest30-21ksoftirqd/220:00:022
38501899864,61cyclictest449335-21ssh23:10:030
38501899864,61cyclictest449335-21ssh23:10:020
385018998330,48cyclictest490486-21ssh00:35:160
385018998323,56cyclictest431359-21ssh22:33:100
38501899821,4cyclictest468290-21sshd23:49:430
385018998128,36cyclictest445454-21ssh23:02:080
385018998121,56cyclictest401644-21ssh21:31:220
385018997918,56cyclictest16850irq/134-800000022:26:460
38501899791,48cyclictest424970-21munin-run22:20:010
38501899791,48cyclictest424970-21munin-run22:20:010
385018997830,44cyclictest392305-21ssh21:11:590
385018997830,44cyclictest392305-21ssh21:11:590
385018997824,37cyclictest0-21swapper/022:59:360
385018997824,37cyclictest0-21swapper/022:59:360
38501899781,46cyclictest439221-21ssh22:49:220
38501899781,41cyclictest1-21systemd23:15:000
385018997763,8cyclictest385020-21cat19:10:010
385018997711,43cyclictest16850irq/134-800000000:23:320
38502899767,49cyclictest446864-21ssh23:05:012
38501899761,71cyclictest394090-21sh21:15:340
385018997532,38cyclictest16950irq/224-800000023:41:430
38501899751,58cyclictest472781-21ssh23:59:060
38501899751,58cyclictest472781-21ssh23:59:060
385018997432,38cyclictest452241-21ssh23:16:060
385018997424,47cyclictest475506-21sshd00:04:430
38501899742,40cyclictest3467602-21systemd-journal00:33:070
38501899741,68cyclictest461450-21ssh23:35:110
38501899741,42cyclictest454935-21sshd23:21:430
385018997320,48cyclictest16850irq/134-800000023:51:380
385018997317,52cyclictest16850irq/134-800000021:55:170
385018997317,52cyclictest16850irq/134-800000021:55:160
385018997316,52cyclictest16850irq/134-800000023:27:110
385018997210,51cyclictest16850irq/134-800000000:15:130
385024997124,8cyclictest400844-21cron21:30:001
385018997134,32cyclictest16850irq/134-800000022:40:010
385018997134,32cyclictest16850irq/134-800000022:40:000
385018997131,36cyclictest411939-21ssh21:52:500
385018997112,42cyclictest487432-21ssh00:29:200
38501899701,64cyclictest386509-21sh19:40:000
38501899701,2cyclictest0-21swapper/022:06:250
385018996930,23cyclictest409004-21ssh21:46:280
38501899692,62cyclictest387258-21sshd19:50:430
385018996926,26cyclictest440765-21ssh22:52:240
385024996833,32cyclictest23-21ksoftirqd/121:32:141
38501899672,30cyclictest397225-21ssh21:22:120
385018996713,45cyclictest16850irq/134-800000021:36:080
38501899662,60cyclictest460365-21ssh23:33:110
38501899661,60cyclictest386764-21sshd19:42:430
38501899661,2cyclictest0-21swapper/021:42:160
38503299653,13cyclictest403256-21munin-run21:35:003
385032996528,33cyclictest491589-21sh00:37:573
385024996530,30cyclictest3467602-21systemd-journal23:11:431
38501899651,5cyclictest0-21swapper/022:40:440
38501899651,2cyclictest0-21swapper/000:13:570
385032996424,36cyclictest37-21ksoftirqd/322:11:313
385024996423,10cyclictest23-21ksoftirqd/123:55:461
385024996423,10cyclictest23-21ksoftirqd/123:55:461
38501899643,57cyclictest477487-21sshd00:08:430
38501899643,57cyclictest477487-21sshd00:08:430
38501899642,57cyclictest0-21swapper/020:45:430
38501899642,57cyclictest0-21swapper/020:45:430
385018996411,44cyclictest16850irq/134-800000021:26:330
385024996325,10cyclictest466044-21diskmemload23:45:001
385024996318,10cyclictest492585-21systemd-run00:40:011
385024996315,12cyclictest3467602-21systemd-journal23:10:021
385024996315,12cyclictest3467602-21systemd-journal23:10:011
38501899633,31cyclictest389337-21sshd20:30:430
38502499625,18cyclictest490149-21munin-plugin-st00:35:011
385024996223,10cyclictest23-21ksoftirqd/122:50:021
385018996228,30cyclictest416024-21rm22:01:010
38501899621,56cyclictest420624-21sshd22:10:440
38503299614,15cyclictest3467602-21systemd-journal00:20:023
385032996123,35cyclictest37-21ksoftirqd/321:25:483
38501899612,54cyclictest385226-21munin-run19:15:010
38501899612,54cyclictest385226-21munin-run19:15:010
38501899612,52cyclictest275-21dbus-daemon20:00:010
38503299607,47cyclictest393615-21munin-run21:15:003
38503299607,47cyclictest393615-21munin-run21:15:003
385032996031,25cyclictest433821-21sh22:38:183
385032996031,25cyclictest433821-21sh22:38:183
385032996022,34cyclictest37-21ksoftirqd/321:37:173
38503299602,19cyclictest424950-21sendmail-msp22:20:003
38503299602,19cyclictest424950-21sendmail-msp22:20:003
385032996020,35cyclictest437761-21rm22:46:153
385024996026,12cyclictest1-21systemd22:15:011
385024996024,5cyclictest473627-21sh00:00:191
385018996029,26cyclictest16950irq/224-800000021:10:000
38501899601,4cyclictest0-21swapper/022:23:550
385032995917,37cyclictest275-21dbus-daemon20:10:013
385032995913,41cyclictest3467602-21systemd-journal20:03:433
38503299583,14cyclictest477771-21ssh00:09:183
38503299583,14cyclictest477771-21ssh00:09:183
38503299583,14cyclictest432181-21munin-plugin-st22:34:593
385032995828,26cyclictest37-21ksoftirqd/323:06:003
385032995828,26cyclictest37-21ksoftirqd/323:05:593
38503299582,19cyclictest442917-21ssh22:56:523
38503299582,19cyclictest442917-21ssh22:56:513
38503299582,14cyclictest415344-21munin-run22:00:013
38503299582,14cyclictest415344-21munin-run22:00:013
385032995820,34cyclictest466229-21date23:45:033
385032995812,41cyclictest0-21swapper/321:21:243
385024995828,10cyclictest23-21ksoftirqd/122:53:301
385024995827,5cyclictest413565-21sh21:55:571
385024995827,5cyclictest413565-21sh21:55:571
385024995827,4cyclictest444802-21sh23:00:401
385024995821,11cyclictest23-21ksoftirqd/123:39:491
38503299572,16cyclictest385931-21latency_hist19:25:023
385032995711,40cyclictest275-21dbus-daemon00:35:033
385024995722,10cyclictest23-21ksoftirqd/100:26:271
385024995721,12cyclictest23-21ksoftirqd/121:54:401
38501899571,2cyclictest0-21swapper/020:55:000
38503299563,16cyclictest48-21kcompactd023:15:273
385032995629,24cyclictest37-21ksoftirqd/300:10:503
38503299562,16cyclictest1-21systemd00:30:013
385032995621,30cyclictest472589-21sh23:58:333
385032995621,30cyclictest472589-21sh23:58:323
38503299561,7cyclictest387281-21cron19:55:003
38503299561,17cyclictest425436-21ssh22:20:343
38503299561,11cyclictest469695-21ssh23:52:303
385024995627,5cyclictest389630-21munin-plugin-st20:40:001
385018995627,26cyclictest0-21swapper/020:20:000
38501899562,49cyclictest387005-21sshd19:45:430
385018995621,30cyclictest1-21systemd19:25:010
38501899561,2cyclictest0-21swapper/020:45:010
385032995531,20cyclictest416845-21sh22:02:543
385032995523,27cyclictest395255-21sh21:18:163
38503299552,19cyclictest418492-21ssh22:06:123
385032995521,30cyclictest437157-21latency_hist22:45:013
385032995518,34cyclictest37-21ksoftirqd/323:04:593
385032995510,9cyclictest459305-1kworker/u9:223:37:423
385028995547,5cyclictest389135-21kworker/2:223:55:582
385028995547,5cyclictest389135-21kworker/2:223:55:582
385028995545,6cyclictest389135-21kworker/2:200:31:232
385028995545,6cyclictest389135-21kworker/2:200:10:002
385028995545,6cyclictest389135-21kworker/2:200:10:002
385028995545,6cyclictest377258-21kworker/2:123:40:142
385024995523,8cyclictest23-21ksoftirqd/122:31:181
385024995521,9cyclictest23-21ksoftirqd/123:15:091
385024995520,11cyclictest408182-21cat21:45:031
38501899552,49cyclictest388327-21munin-run20:15:010
38501899552,49cyclictest386019-21munin-run19:30:010
38501899552,48cyclictest390938-21munin-run21:05:020
38503299542,8cyclictest475671-21systemd-run00:05:003
38503299542,10cyclictest389361-21modprobe20:35:003
38503299541,15cyclictest454247-21ssh23:20:043
38503299541,14cyclictest439953-21ssh22:50:343
38503299541,10cyclictest483246-21ssh00:20:193
38503299541,10cyclictest410756-21ssh21:50:043
385028995446,5cyclictest377258-21kworker/2:122:47:072
385028995446,5cyclictest377258-21kworker/2:122:25:132
385028995413,37cyclictest437213-21ssh22:45:032
38501899541,6cyclictest389587-21sshd20:35:430
38501899541,48cyclictest389123-21munin-run20:30:000
38501899541,47cyclictest390623-21sshd20:55:430
38503299533,6cyclictest1-21systemd19:40:013
385032995329,20cyclictest387011-21sshd19:46:553
38503299532,7cyclictest450272-21ssh23:12:083
38503299532,6cyclictest389136-21unin-run20:30:003
38503299532,16cyclictest458046-21ssh23:28:213
38503299532,15cyclictest429399-21sh22:29:143
38503299531,9cyclictest459364-21ssh23:30:463
38503299531,7cyclictest467156-21ssh23:47:103
38503299531,11cyclictest364-21snmpd21:47:033
385028995345,5cyclictest389135-21kworker/2:223:23:472
385028995345,5cyclictest389135-21kworker/2:223:11:332
385028995345,5cyclictest377258-21kworker/2:121:56:142
385028995345,5cyclictest377258-21kworker/2:121:56:132
385028995345,5cyclictest377258-21kworker/2:120:25:032
385028995345,5cyclictest377258-21kworker/2:120:25:022
385028995344,6cyclictest377258-21kworker/2:122:04:592
385028995344,6cyclictest377258-21kworker/2:121:42:142
385028995343,6cyclictest377258-21kworker/2:121:48:342
385024995324,7cyclictest23-21ksoftirqd/122:01:091
385024995323,8cyclictest23-21ksoftirqd/121:12:031
385024995323,8cyclictest23-21ksoftirqd/121:12:031
385024995322,9cyclictest23-21ksoftirqd/100:16:411
38501899532,47cyclictest388869-21munin-run20:25:010
38501899532,47cyclictest388869-21munin-run20:25:010
38503299522,6cyclictest390668-21munin-run21:00:003
38503299522,15cyclictest386007-21munin-run19:30:013
38503299521,18cyclictest385209-21sh19:15:003
38503299521,18cyclictest385209-21sh19:15:003
38503299521,11cyclictest407989-21ssh21:44:453
385028995244,5cyclictest389135-21kworker/2:223:49:522
385028995243,6cyclictest389135-21kworker/2:221:18:242
385028995242,6cyclictest389135-21kworker/2:221:10:372
385028995242,6cyclictest389135-21kworker/2:221:10:372
385028995242,6cyclictest377258-21kworker/2:122:07:532
385028995242,6cyclictest377258-21kworker/2:121:50:202
385028995242,6cyclictest377258-21kworker/2:121:26:552
385024995224,8cyclictest23-21ksoftirqd/123:32:231
385024995220,7cyclictest3467602-21systemd-journal20:00:001
38503299513,7cyclictest284-21systemd-logind20:20:013
38503299512,7cyclictest0-21swapper/319:20:023
38503299512,18cyclictest390405-21munin-run20:55:003
385028995143,5cyclictest389135-21kworker/2:223:16:252
385028995143,5cyclictest389135-21kworker/2:222:16:362
385028995143,5cyclictest389135-21kworker/2:222:16:362
385028995143,5cyclictest389135-21kworker/2:200:23:062
385028995143,5cyclictest377258-21kworker/2:122:11:342
385028995142,6cyclictest389135-21kworker/2:223:27:302
385028995142,6cyclictest389135-21kworker/2:221:36:522
385028995142,6cyclictest389135-21kworker/2:221:20:532
385028995142,6cyclictest389135-21kworker/2:200:01:072
385028995142,5cyclictest468477-21kworker/2:000:36:362
385028995142,5cyclictest377258-21kworker/2:122:56:202
385028995142,5cyclictest377258-21kworker/2:122:56:192
385028995142,5cyclictest377258-21kworker/2:122:51:052
385028995140,6cyclictest389135-21kworker/2:223:52:042
38501899512,44cyclictest364-21snmpd20:01:250
38501899511,46cyclictest364-21snmpd19:15:460
38503299502,7cyclictest275-21dbus-daemon19:35:003
38503299501,9cyclictest1-21systemd20:25:013
38503299501,9cyclictest1-21systemd20:25:013
385028995042,5cyclictest389135-21kworker/2:200:17:292
385028995042,5cyclictest377258-21kworker/2:123:35:372
385028995041,6cyclictest377258-21kworker/2:123:34:492
385028995040,6cyclictest377258-21kworker/2:122:33:092
385028995040,6cyclictest377258-21kworker/2:120:20:002
385028995031,14cyclictest434575-21cron22:40:002
385028995031,14cyclictest434575-21cron22:40:002
38501899502,44cyclictest388052-21sshd20:07:430
38503299494,3cyclictest0-21swapper/321:10:003
38503299493,8cyclictest390997-21cat21:05:023
38503299491,14cyclictest1-21systemd20:00:013
385028994941,5cyclictest389135-21kworker/2:200:25:202
385028994941,5cyclictest377258-21kworker/2:122:22:252
385028994940,5cyclictest389135-21kworker/2:223:05:502
385028994940,5cyclictest389135-21kworker/2:223:05:492
385028994940,5cyclictest389135-21kworker/2:200:11:452
385024994921,9cyclictest23-21ksoftirqd/122:39:251
385024994921,9cyclictest23-21ksoftirqd/122:39:241
38503299482,8cyclictest390119-21sshd20:45:433
38503299482,8cyclictest390119-21sshd20:45:433
38503299482,6cyclictest3467602-21systemd-journal20:45:013
38503299482,16cyclictest386771-21sshd19:43:443
385028994839,6cyclictest389135-21kworker/2:221:30:512
385028994839,6cyclictest373808-21kworker/2:319:50:272
38502499482,24cyclictest398482-21ssh21:25:001
38503299473,6cyclictest364-21snmpd20:11:423
385028994739,5cyclictest373808-21kworker/2:319:37:462
385028994739,5cyclictest373808-21kworker/2:319:18:412
385028994738,6cyclictest389135-21kworker/2:221:00:402
385024994712,6cyclictest131rcu_preempt22:23:431
38501899471,42cyclictest386229-21sshd19:31:520
385028994638,5cyclictest377258-21kworker/2:120:14:262
385028994638,5cyclictest377258-21kworker/2:120:10:022
385028994638,5cyclictest373808-21kworker/2:319:29:072
385024994632,10cyclictest3467602-21systemd-journal23:23:431
385024994610,4cyclictest346771850lldpd00:23:471
385028994538,5cyclictest377258-21kworker/2:120:01:242
385028994538,4cyclictest389135-21kworker/2:220:58:542
385028994537,5cyclictest377258-21kworker/2:119:12:012
385028994537,5cyclictest377258-21kworker/2:119:12:002
385028994537,5cyclictest373808-21kworker/2:319:24:182
385028994536,6cyclictest389135-21kworker/2:220:51:112
385028994536,6cyclictest389135-21kworker/2:220:40:042
385028994532,9cyclictest3467602-21systemd-journal20:36:322
38503299442,14cyclictest385101-21latency_hist19:10:023
38503299441,12cyclictest364-21snmpd20:38:283
385028994437,4cyclictest377258-21kworker/2:120:31:292
385028994436,5cyclictest389135-21kworker/2:221:07:362
385028994435,6cyclictest373808-21kworker/2:319:40:472
385024994435,7cyclictest23-21ksoftirqd/119:15:021
385024994435,7cyclictest23-21ksoftirqd/119:15:021
385028994336,5cyclictest377258-21kworker/2:120:27:082
385028994336,4cyclictest389135-21kworker/2:220:45:132
385028994336,4cyclictest389135-21kworker/2:220:45:132
385028994234,5cyclictest373808-21kworker/2:319:32:072
385028994233,5cyclictest373808-21kworker/2:319:47:132
385024994229,9cyclictest470800-21cron23:55:001
385024994225,12cyclictest275-21dbus-daemon21:05:011
385024994127,11cyclictest23-21ksoftirqd/119:45:011
385024994126,9cyclictest309-21in:imuxsock20:25:011
385024994126,9cyclictest309-21in:imuxsock20:25:001
385024994124,12cyclictest1-21systemd00:10:021
385024994124,12cyclictest1-21systemd00:10:011
385024994123,13cyclictest309-21in:imuxsock20:30:021
385024994029,8cyclictest1-21systemd19:40:011
385024994028,9cyclictest23-21ksoftirqd/121:10:011
385024994027,10cyclictest23-21ksoftirqd/119:50:001
385024994023,11cyclictest309-21in:imuxsock22:55:021
385024994023,11cyclictest309-21in:imuxsock22:55:021
385024994011,7cyclictest131rcu_preempt22:41:191
38502499395,4cyclictest131rcu_preempt22:06:261
385024993928,9cyclictest23-21ksoftirqd/122:17:431
385024993928,9cyclictest23-21ksoftirqd/122:17:431
385024993927,9cyclictest23-21ksoftirqd/121:00:021
385024993927,10cyclictest395465-21sshd21:18:431
385024993823,11cyclictest386285-21sed19:35:011
385024993727,8cyclictest23-21ksoftirqd/123:30:001
385024993726,8cyclictest23-21ksoftirqd/122:26:501
38502499372,5cyclictest131rcu_preempt00:15:001
385024993725,9cyclictest23-21ksoftirqd/121:47:321
385024993724,10cyclictest388269-21sed20:10:021
385024993719,14cyclictest1-21systemd19:30:011
385024993626,6cyclictest388549-21cron20:17:001
385024993622,10cyclictest23-21ksoftirqd/120:03:111
385024993621,10cyclictest309-21in:imuxsock20:55:001
385024993620,13cyclictest275-21dbus-daemon21:40:011
385024993528,4cyclictest389901-21munin-plugin-st20:45:001
38502499352,4cyclictest131rcu_preempt19:10:021
385024993523,10cyclictest23-21ksoftirqd/123:49:011
385028993423,7cyclictest0-21swapper/219:10:012
385024993421,10cyclictest23-21ksoftirqd/120:29:011
38502499341,4cyclictest131rcu_preempt20:50:001
38502499341,4cyclictest131rcu_preempt20:50:001
385024993322,9cyclictest23-21ksoftirqd/119:51:521
385024993222,5cyclictest388313-21cron20:15:001
38502499314,20cyclictest385461-21cron19:20:001
385024993023,4cyclictest385751-21latency_hist19:25:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional