You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-17 - 20:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Wed Jun 17, 2026 12:45:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
306954899912,4cyclictest12-21ksoftirqd/008:10:020
3069553998778,6cyclictest23-21ksoftirqd/107:25:021
3069553998224,20cyclictest3112200-21ssh10:30:441
306955399814,5cyclictest131rcu_preempt11:10:021
3069548998121,37cyclictest12-21ksoftirqd/010:10:020
3069553998024,15cyclictest3165738-21ssh12:29:431
3069553998021,24cyclictest3151050-21rm11:56:521
306954899791,2cyclictest0-21swapper/007:55:020
3069553997828,14cyclictest3157123-21ssh12:10:061
3069553997724,15cyclictest3138688-21ssh11:29:471
3069553997722,21cyclictest3170213-21ssh12:39:451
3069553997720,16cyclictest3116534-21ssh10:40:061
3069553997625,18cyclictest3088485-21ssh09:38:081
3069553997620,22cyclictest3140141-21ssh11:32:391
3069553997620,22cyclictest3140141-21ssh11:32:391
3069553997620,21cyclictest3107850-21ssh10:20:591
3069553997620,18cyclictest3159430-21ssh12:15:111
3069553997620,18cyclictest3159430-21ssh12:15:101
3069553997524,15cyclictest3149499-21ssh11:53:381
3069553997524,15cyclictest3149499-21ssh11:53:381
3069553997524,15cyclictest3079175-21ssh09:17:241
3069553997522,18cyclictest3126927-21rm11:03:311
3069553997428,14cyclictest3141451-21sh11:35:211
3069553997424,14cyclictest3086015-21rm09:32:381
3069553997326,15cyclictest3102328-21ssh10:09:001
3069553997323,14cyclictest3084433-21ssh09:29:161
3069553997320,19cyclictest3156707-21ssh12:09:411
3069553997227,15cyclictest3124744-21ssh10:58:431
3069553997223,14cyclictest3123007-21ssh10:54:521
3069553997122,19cyclictest3105439-21ssh10:15:441
3069553997121,20cyclictest3080852-21ssh09:20:551
3069553997120,17cyclictest3115228-21ssh10:37:371
3069553997120,15cyclictest3133314-21ssh11:17:321
3069553997119,15cyclictest3099713-21ssh10:02:571
3069562997017,39cyclictest3150132-21ssh11:55:023
3069562997017,39cyclictest3150132-21ssh11:55:023
3069553997021,16cyclictest3094795-21ssh09:51:551
3069553996924,8cyclictest3136406-21ssh11:24:381
3069553996924,8cyclictest3136406-21ssh11:24:381
3069553996923,16cyclictest3119344-21sh10:46:281
3069553996922,18cyclictest3093635-21ssh09:49:401
3069553996921,16cyclictest3071188-21latency_hist07:40:021
3069553996920,14cyclictest3167894-21ssh12:34:331
3069553996828,14cyclictest3097568-21ssh09:58:131
3069553996824,13cyclictest3131636-21sh11:14:001
3069553996824,13cyclictest3131636-21sh11:14:001
3069553996821,15cyclictest3091211-21ssh09:44:191
3069553996728,12cyclictest3111260-21ssh10:28:531
3069553996722,14cyclictest3072614-21latency_hist08:10:011
3069553996620,15cyclictest3098769-21kworker/u8:010:14:071
3069553996518,15cyclictest3074811-21cat08:50:011
3069553996422,14cyclictest3163289-21ssh12:24:131
3069553996420,14cyclictest3075831-21latency_hist09:10:021
3069553996417,13cyclictest3146649-21ssh11:47:031
3069553996320,14cyclictest3143743-21rm11:40:211
3069553996320,14cyclictest3143743-21rm11:40:211
3069553996317,13cyclictest3154442-21ssh12:04:381
3069553996317,13cyclictest3154442-21ssh12:04:381
3069548996330,15cyclictest3091923-21ssh09:45:250
3069553996245,13cyclictest3077801-21ssh09:14:371
3069553996039,16cyclictest3073647-21cron08:30:011
3069553995921,13cyclictest364-21snmpd08:18:371
3069548995930,13cyclictest3135289-21ssh11:21:510
3069548995930,13cyclictest3135289-21ssh11:21:510
3069548995922,33cyclictest12-21ksoftirqd/012:22:260
3069553995845,9cyclictest3070046-21munin-run07:20:011
3069553995844,11cyclictest3075178-21sshd08:58:501
3069553995841,13cyclictest3467602-21systemd-journal07:50:091
3069548995816,35cyclictest16850irq/134-800000012:00:010
3069548995812,15cyclictest131rcu_preempt12:29:100
3069548995811,24cyclictest3071012-21kworker/u8:308:00:030
3069558995735,17cyclictest275-21dbus-daemon09:45:012
3069558995646,6cyclictest3072897-21kworker/2:009:14:572
3069558995644,7cyclictest3074700-21grep08:50:002
3069548995652,2cyclictest0-21swapper/012:35:010
306954899562,48cyclictest1-21systemd09:00:010
3069558995546,6cyclictest3072897-21kworker/2:011:38:242
3069558995545,6cyclictest3074710-21kworker/2:210:48:552
3069558995545,5cyclictest3074710-21kworker/2:210:53:482
3069553995542,9cyclictest364-21snmpd08:33:401
3069553995531,19cyclictest3069733-21cron07:15:001
3069548995519,31cyclictest3083235-21sh09:26:160
306954899551,37cyclictest131rcu_preempt11:35:020
306954899551,37cyclictest131rcu_preempt11:35:020
3069562995432,18cyclictest3071093-21latency_hist07:40:003
3069558995445,6cyclictest3074710-21kworker/2:209:26:162
3069558995445,5cyclictest3072897-21kworker/2:011:27:572
3069558995444,6cyclictest3074710-21kworker/2:212:31:022
3069558995433,14cyclictest3467602-21systemd-journal11:00:022
3069553995422,8cyclictest3073394-21sshd08:24:161
3069548995424,24cyclictest16850irq/134-800000010:25:000
3069562995336,13cyclictest275-21dbus-daemon07:20:003
306956299533,26cyclictest131rcu_preempt11:20:023
3069558995345,5cyclictest3072897-21kworker/2:010:14:162
3069558995344,6cyclictest3072897-21kworker/2:011:49:412
3069558995344,6cyclictest3072897-21kworker/2:010:38:352
3069558995343,6cyclictest3074710-21kworker/2:209:52:062
3069558995337,7cyclictest3072897-21kworker/2:012:07:512
3069558995335,13cyclictest1-21systemd09:20:012
3069553995341,9cyclictest3072064-21sshd07:58:371
3069553995340,9cyclictest48-21kcompactd009:04:471
3069553995340,10cyclictest364-21snmpd07:43:301
3069553995339,11cyclictest3070804-21munin-run07:35:011
3069553995336,13cyclictest364-21snmpd07:48:021
3069548995345,4cyclictest16950irq/224-800000010:40:010
3069548995325,13cyclictest12-21ksoftirqd/011:00:000
3069548995317,32cyclictest3074695-21latency_hist08:50:020
306954899531,37cyclictest3158583-21ssh12:13:480
306956299526,8cyclictest37-21ksoftirqd/309:21:403
3069558995244,5cyclictest3074710-21kworker/2:211:18:272
3069558995244,5cyclictest3074710-21kworker/2:209:48:432
3069558995244,4cyclictest3072897-21kworker/2:011:30:512
3069558995244,4cyclictest3072897-21kworker/2:011:30:512
3069558995243,6cyclictest3072897-21kworker/2:010:29:302
3069558995243,5cyclictest3072897-21kworker/2:010:17:302
3069558995235,12cyclictest1-21systemd08:25:012
3069553995240,9cyclictest3074135-21sshd08:36:161
3069553995234,14cyclictest364-21snmpd08:02:141
3069548995242,7cyclictest0-21swapper/009:10:010
3069548995230,15cyclictest275-21dbus-daemon10:30:000
3069548995228,10cyclictest3086068-21ssh09:32:450
3069548995225,12cyclictest3169593-21ssh12:38:120
3069548995224,23cyclictest3078151-21ssh09:15:020
3069562995112,34cyclictest3072381-21unin-run08:05:013
3069558995143,5cyclictest3074710-21kworker/2:209:32:592
3069558995143,5cyclictest3072897-21kworker/2:012:14:262
3069558995143,5cyclictest3072897-21kworker/2:011:53:132
3069558995143,5cyclictest3072897-21kworker/2:011:53:132
3069558995142,6cyclictest3074710-21kworker/2:209:59:432
3069558995142,6cyclictest3072897-21kworker/2:012:02:432
3069558995142,6cyclictest3072897-21kworker/2:012:02:422
3069558995142,6cyclictest3072897-21kworker/2:010:20:462
3069558995142,5cyclictest3074710-21kworker/2:212:26:402
3069558995142,5cyclictest3072897-21kworker/2:011:42:102
3069558995142,5cyclictest3072897-21kworker/2:011:42:102
3069558995141,7cyclictest3060859-21kworker/2:108:29:302
3069558995135,12cyclictest3170356-21systemctl12:40:002
3069558995129,13cyclictest3160753-21ssh12:18:302
3069558995129,13cyclictest3160753-21ssh12:18:302
306955399517,26cyclictest3072359-21kworker/u8:108:55:021
3069553995131,15cyclictest3070544-21sh07:30:001
306954899518,37cyclictest16850irq/134-800000011:15:020
306954899518,37cyclictest16850irq/134-800000011:15:020
3069548995127,9cyclictest3142807-21ssh11:38:400
306954899512,44cyclictest3150085-21unin-run11:55:010
306954899512,44cyclictest3150085-21unin-run11:55:010
3069548995124,12cyclictest3127035-21ssh11:03:470
3069548995116,31cyclictest16850irq/134-800000009:35:070
3069548995115,7cyclictest12-21ksoftirqd/010:35:030
306956299501,11cyclictest3166991-21ssh12:32:193
3069558995042,5cyclictest3074710-21kworker/2:211:14:042
3069558995042,5cyclictest3074710-21kworker/2:211:14:042
3069558995042,5cyclictest3074710-21kworker/2:210:56:152
3069558995042,5cyclictest3074710-21kworker/2:210:00:072
3069558995042,5cyclictest3074710-21kworker/2:209:36:572
3069558995042,5cyclictest3072897-21kworker/2:010:44:462
3069558995042,5cyclictest3072897-21kworker/2:009:24:202
3069558995041,6cyclictest3072897-21kworker/2:011:58:432
3069558995041,6cyclictest3072897-21kworker/2:010:33:352
3069558995031,13cyclictest275-21dbus-daemon11:10:002
3069553995037,10cyclictest364-21snmpd08:13:361
306954899503,7cyclictest3467602-21systemd-journal11:45:000
306954899503,7cyclictest3467602-21systemd-journal11:45:000
3069548995031,14cyclictest1-21systemd10:00:010
306954899502,15cyclictest3134332-21systemd-run11:20:010
3069548995015,5cyclictest16850irq/134-800000010:20:010
3069558994941,5cyclictest3074710-21kworker/2:211:20:032
3069558994941,5cyclictest3074710-21kworker/2:211:20:032
3069558994941,5cyclictest3072897-21kworker/2:009:03:062
3069558994941,5cyclictest3072897-21kworker/2:008:44:172
3069558994941,5cyclictest3072897-21kworker/2:008:38:412
3069558994940,6cyclictest3072897-21kworker/2:010:06:412
3069558994940,5cyclictest3074710-21kworker/2:212:20:572
3069558994939,6cyclictest3060859-21kworker/2:108:15:082
306954899495,33cyclictest2639831-1kworker/0:2H09:20:020
306954899491,3cyclictest131rcu_preempt12:01:170
306954899491,3cyclictest131rcu_preempt12:01:170
306956299483,6cyclictest3092229-21ssh09:46:153
3069562994823,5cyclictest37-21ksoftirqd/312:10:463
3069558994840,5cyclictest3060859-21kworker/2:108:05:162
3069558994840,5cyclictest3060859-21kworker/2:107:14:172
3069558994839,6cyclictest3070044-21kworker/2:307:50:182
3069553994835,9cyclictest48-21kcompactd008:44:491
3069548994830,15cyclictest16850irq/134-800000011:50:000
3069548994824,7cyclictest12-21ksoftirqd/009:55:020
306954899482,12cyclictest3074217-21wc08:40:020
3069562994730,13cyclictest275-21dbus-daemon11:15:013
3069562994730,13cyclictest275-21dbus-daemon11:15:003
3069558994739,5cyclictest3060859-21kworker/2:107:27:422
3069562994629,11cyclictest3467602-21systemd-journal10:15:003
3069562994627,14cyclictest1-21systemd11:30:003
3069558994638,5cyclictest3072897-21kworker/2:009:06:472
3069558994638,5cyclictest3070044-21kworker/2:307:43:392
3069558994629,12cyclictest30-21ksoftirqd/208:13:152
306954899467,26cyclictest3069551-21latency_hist07:10:010
3069548994638,4cyclictest16950irq/224-800000011:30:000
3069548994631,11cyclictest275-21dbus-daemon09:05:010
3069548994627,15cyclictest3103335-21ssh10:10:560
306954899462,38cyclictest275-21dbus-daemon10:05:010
3069548994610,20cyclictest16850irq/134-800000012:09:010
3069558994537,5cyclictest3070044-21kworker/2:307:58:262
3069558994537,5cyclictest3070044-21kworker/2:307:38:292
3069558994537,5cyclictest3070044-21kworker/2:307:33:512
3069558994537,5cyclictest3060859-21kworker/2:107:16:452
3069558994537,4cyclictest3072897-21kworker/2:008:50:172
3069558994536,6cyclictest3072897-21kworker/2:008:32:262
3069548994531,11cyclictest3467602-21systemd-journal10:45:160
3069548994518,12cyclictest3128569-21ssh11:07:010
3069548994516,26cyclictest12-21ksoftirqd/009:22:430
3069548994511,31cyclictest16850irq/134-800000012:18:410
3069548994511,31cyclictest16850irq/134-800000012:18:410
3069562994430,10cyclictest275-21dbus-daemon07:30:013
3069562994429,11cyclictest1-21systemd08:20:003
306956299442,12cyclictest3105605-21ssh10:16:083
3069558994437,4cyclictest3072897-21kworker/2:008:55:512
3069558994437,4cyclictest3060859-21kworker/2:107:20:192
3069558994436,5cyclictest3070044-21kworker/2:308:03:502
3069558994436,5cyclictest3070044-21kworker/2:307:47:472
306954899449,31cyclictest16850irq/134-800000010:50:480
306954899448,21cyclictest16850irq/134-800000010:44:370
306956299439,8cyclictest0-21swapper/312:05:593
306956299433,3cyclictest131rcu_preempt09:35:263
306954899437,21cyclictest16850irq/134-800000009:40:190
3069548994327,11cyclictest3467602-21systemd-journal08:20:000
3069548994320,20cyclictest12-21ksoftirqd/007:23:250
3069562994232,7cyclictest37-21ksoftirqd/308:00:023
3069562994227,11cyclictest37-21ksoftirqd/309:14:123
3069562994225,13cyclictest37-21ksoftirqd/309:20:013
306956299422,13cyclictest3153561-21ssh12:02:303
306956299422,13cyclictest3153561-21ssh12:02:303
306956299421,10cyclictest3117506-21ssh10:42:353
3069548994228,10cyclictest3071579-21munin-run07:49:590
3069548994211,27cyclictest0-21swapper/008:04:240
306956299416,7cyclictest37-21ksoftirqd/309:34:533
3069562994132,5cyclictest3073659-21latency_hist08:30:013
3069562994130,8cyclictest3152363-21sendmail-msp12:00:003
3069562994130,8cyclictest275-21dbus-daemon08:15:013
3069562994127,11cyclictest1-21systemd10:10:013
306956299412,21cyclictest361rcuc/310:25:023
306956299412,12cyclictest3084201-21ssh09:28:373
306956299411,3cyclictest131rcu_preempt12:35:193
306956299411,10cyclictest3127366-21ssh11:04:323
306954899413,3cyclictest131rcu_preempt07:40:010
3069548994124,13cyclictest3072843-21sshd08:12:160
306956299405,3cyclictest131rcu_preempt11:41:473
306956299405,3cyclictest131rcu_preempt11:41:473
306956299403,9cyclictest0-21swapper/308:51:283
3069562994031,6cyclictest37-21ksoftirqd/310:55:013
3069562994028,9cyclictest37-21ksoftirqd/309:08:473
3069562994028,9cyclictest3467602-21systemd-journal12:25:003
3069562994027,10cyclictest3074224-21sed08:40:023
3069562994025,11cyclictest3467602-21systemd-journal07:50:003
3069562994023,14cyclictest3100638-21latency_hist10:05:023
3069562994021,13cyclictest275-21dbus-daemon08:25:013
3069548994027,9cyclictest3070503-21sshd07:25:160
306956299393,5cyclictest3111763-21kworker/3:011:35:103
3069562993929,8cyclictest37-21ksoftirqd/310:38:283
3069562993929,8cyclictest37-21ksoftirqd/310:26:153
3069562993929,6cyclictest3125313-21cron11:00:003
3069562993926,9cyclictest37-21ksoftirqd/309:50:033
3069562993926,10cyclictest37-21ksoftirqd/308:59:593
3069562993923,13cyclictest37-21ksoftirqd/311:50:003
3069562993923,12cyclictest3467602-21systemd-journal07:20:033
306956299391,3cyclictest131rcu_preempt10:47:383
306954899392,10cyclictest3073459-21latency_hist08:25:020
306954899391,4cyclictest131rcu_preempt07:35:010
3069548993912,23cyclictest3070009-21latency_hist07:20:020
3069562993828,6cyclictest3070811-21latency_hist07:35:023
3069562993827,8cyclictest37-21ksoftirqd/311:32:523
3069562993827,8cyclictest37-21ksoftirqd/311:32:523
3069562993825,10cyclictest3114023-21sed10:35:023
3069562993823,11cyclictest37-21ksoftirqd/309:55:503
3069553993827,9cyclictest0-21swapper/107:10:021
3069548993831,4cyclictest16950irq/224-800000007:15:010
306954899381,5cyclictest131rcu_preempt07:45:030
306956299375,3cyclictest131rcu_preempt07:15:003
3069548993730,4cyclictest19350irq/34-mmc108:26:030
3069548993728,6cyclictest19350irq/34-mmc108:50:570
306954899371,32cyclictest0-21swapper/008:30:030
306956299363,3cyclictest131rcu_preempt12:26:213
306956299361,3cyclictest131rcu_preempt11:06:373
3069562993528,4cyclictest3159487-21sh12:15:283
3069562993528,4cyclictest3159487-21sh12:15:283
3069562993523,8cyclictest3072012-21grep07:55:013
3069562993521,10cyclictest275-21dbus-daemon09:45:013
3069562993426,5cyclictest3135603-21sh11:22:343
3069562993426,5cyclictest3135603-21sh11:22:343
306956299341,3cyclictest131rcu_preempt09:05:023
306954899341,29cyclictest0-21swapper/008:42:080
3069562993319,10cyclictest3074895-21date08:50:023
3069562993221,9cyclictest37-21ksoftirqd/308:07:313
306956299318,4cyclictest131rcu_preempt08:32:003
306956299313,3cyclictest131rcu_preempt08:45:023
3069558993122,6cyclictest30-21ksoftirqd/207:10:032
306956299302,3cyclictest131rcu_preempt07:45:013
306956299274,3cyclictest131rcu_preempt07:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional