You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-05 - 21:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Sun Jul 05, 2026 12:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3232499645,9cyclictest131rcu_preempt09:55:120
32324996310,10cyclictest131rcu_preempt12:00:130
3233699611,50cyclictest7825-21ssh09:40:121
3233699595,43cyclictest31335-21ssh11:05:151
3233699591,37cyclictest2642-21apt-get08:20:131
3232499564,37cyclictest24140-21ssh10:25:130
3233699541,48cyclictest736-21apt-get12:00:131
32324995431,11cyclictest12-21ksoftirqd/007:50:110
3233699501,43cyclictest4876-21apt-get07:15:131
3233699481,43cyclictest13470-21apt-get09:05:131
3233699471,39cyclictest20639-21apt-get08:05:111
3233699470,4cyclictest8027-21ssh11:10:121
3233699470,41cyclictest20601-21apt-get11:35:141
3233699462,39cyclictest3782-21apt-get08:55:121
3233699461,41cyclictest27481-21apt-get09:15:141
3233699461,40cyclictest32320-21cyclictest08:40:131
3233699461,40cyclictest25403-21apt-get08:10:131
3233699461,38cyclictest6197-21apt-get07:50:101
3232499454,33cyclictest17023-21ssh11:15:140
3232499451,36cyclictest9708-21ssh12:05:120
3232499451,31cyclictest20606-21ssh11:35:140
32352994420,16cyclictest0-21swapper/311:45:143
3234499441,9cyclictest29543-21ssh10:10:132
3234499441,9cyclictest29543-21ssh10:10:122
3234499441,10cyclictest27346irq/40-dwc2_hso09:30:132
3233699441,39cyclictest0-21swapper/110:25:141
3233699441,38cyclictest15130-21apt-get10:20:141
3232499441,38cyclictest6097-21ssh10:15:120
3234499431,10cyclictest0-21swapper/208:35:142
3233699432,37cyclictest0-21swapper/109:50:121
32336994318,18cyclictest21-21ksoftirqd/108:15:121
3233699426,28cyclictest31181-21ssh09:35:111
3233699422,35cyclictest22148-21apt-get09:30:141
3233699422,32cyclictest2555-21apt-get11:25:131
3233699421,36cyclictest27608-21apt-get12:15:141
3233699421,36cyclictest11004-21apt-get07:55:141
3233699421,36cyclictest11004-21apt-get07:55:131
3233699421,25cyclictest0-21swapper/110:39:121
3232499422,24cyclictest18739-21ssh12:10:160
3233699418,24cyclictest21-21ksoftirqd/107:10:121
3233699418,24cyclictest21-21ksoftirqd/107:10:111
3232499413,23cyclictest13097-21ssh09:25:130
3233699401,35cyclictest18748-21apt-get10:40:141
3233699401,35cyclictest13087-21apt-get09:25:121
3233699401,35cyclictest0-21swapper/110:50:141
3233699401,34cyclictest4014-21apt-get09:20:131
3233699401,34cyclictest4014-21apt-get09:20:121
3233699400,34cyclictest16872-21apt-get09:45:131
3234499393,15cyclictest27346irq/40-dwc2_hso12:20:122
3234499391,14cyclictest32091-21kthreadcore08:15:212
3233699391,33cyclictest9733-21apt-get07:20:131
3233699391,33cyclictest6080-21apt-get10:15:111
3233699391,23cyclictest26087-21proc_pri11:00:281
3235299382,19cyclictest35966irq/58-eth007:20:143
3234499381,8cyclictest16460-21kthreadcore09:05:232
3234499381,14cyclictest30937-21kthreadcore07:40:222
3234499381,10cyclictest1330-21sed11:40:302
3234499381,10cyclictest1330-21sed11:40:302
3234499380,13cyclictest22779-21open_files07:30:272
3234499380,12cyclictest8765-21kthreadcore10:15:212
3233699381,33cyclictest0-21swapper/111:55:141
3233699381,32cyclictest2371-21ntp_192.168.11412:35:241
3233699380,24cyclictest10302-21rm09:59:001
3234499375,9cyclictest4503-21kthreadcore08:20:192
3234499371,9cyclictest16989-21sendmail_mailqu12:25:272
3234499371,8cyclictest7575-21fschecks_count08:25:152
3234499371,15cyclictest12415-21grep09:00:292
3234499371,11cyclictest15497-21timerandwakeup10:00:292
3233699371,31cyclictest15450-21fschecks_time11:50:151
3232499372,31cyclictest25986-21ssh11:20:130
3235299364,19cyclictest35966irq/58-eth011:25:143
32352993621,9cyclictest35966irq/58-eth011:15:143
3234499364,23cyclictest27712-21ssh10:45:142
3234499361,7cyclictest31376-21latency_hist08:50:022
3234499361,7cyclictest10986-21/usr/sbin/munin07:55:222
3234499361,7cyclictest10986-21/usr/sbin/munin07:55:212
3234499361,14cyclictest12358-21fschecks_count08:30:152
3234499361,11cyclictest11975-21grep09:40:282
3233699362,31cyclictest13184-21apt-get12:25:141
32336993613,14cyclictest21-21ksoftirqd/111:15:121
3233699361,30cyclictest9696-21apt-get12:05:131
3233699361,30cyclictest17011-21apt-get08:35:131
3232499361,4cyclictest131rcu_preempt11:30:140
3235299353,19cyclictest35966irq/58-eth011:35:143
3235299352,20cyclictest35966irq/58-eth009:45:113
3234499356,6cyclictest5901-21kthreadcore10:50:212
3234499351,11cyclictest11732-21ssh11:30:162
3234499351,10cyclictest27890-21sed09:15:182
32344993510,4cyclictest0-21swapper/208:55:282
3233699351,30cyclictest20493-21latency_hist08:05:011
3233699351,30cyclictest16758-21cut07:25:211
3233699351,28cyclictest0-21swapper/112:20:121
3233699350,32cyclictest6208-21apt-get11:45:131
3233699350,22cyclictest5496-21ssh10:14:471
3233699350,22cyclictest5496-21ssh10:14:461
3232499353,23cyclictest27489-21ssh09:15:120
3234499341,8cyclictest27997-21irqcore12:15:172
3234499341,14cyclictest20542-21cut10:40:222
3234499340,8cyclictest20125-21kthreadcore09:45:222
3234499340,6cyclictest18602-21kthreadcore08:00:212
3233699341,30cyclictest6021-21grep07:50:021
3233699341,28cyclictest27687-21ssh10:45:131
3233699340,29cyclictest29619-21ssh11:40:121
3233699340,29cyclictest29619-21ssh11:40:111
3235299332,21cyclictest35966irq/58-eth010:05:133
3234499331,9cyclictest1324-21cut07:10:202
3234499331,9cyclictest1324-21cut07:10:192
3234499331,8cyclictest27873-21ssh11:55:262
3234499331,7cyclictest25384-21ssh09:13:032
3234499331,10cyclictest31796-21irqrtprio12:35:192
3234499330,7cyclictest18728-21ssh10:20:242
3233699339,21cyclictest28761-21cron07:40:021
3233699331,28cyclictest0-21swapper/110:05:131
3233699330,29cyclictest31420-21apt-get08:50:141
3233699330,29cyclictest31420-21apt-get08:50:141
3232499332,23cyclictest29653-21ssh11:40:140
3232499332,23cyclictest29653-21ssh11:40:130
32324993314,16cyclictest12-21ksoftirqd/012:35:120
3232499331,24cyclictest9742-21ssh10:35:130
3235299322,14cyclictest35966irq/58-eth011:40:123
3235299322,14cyclictest35966irq/58-eth011:40:113
3235299321,18cyclictest35966irq/58-eth012:10:153
3234499322,10cyclictest26079-21wc11:00:292
3234499321,9cyclictest27274-21ps08:10:192
3234499321,8cyclictest2910-21kthreadcore07:45:202
3234499321,8cyclictest25930-21kthreadcore07:35:222
3234499321,8cyclictest11952-21switchtime11:10:292
3234499321,6cyclictest27858-21ntp_kernel_err10:25:302
3234499321,15cyclictest25995-21ssh11:20:132
32344993211,3cyclictest4680-21ssh12:00:292
3234499321,11cyclictest9704-21proc_pri07:50:242
3233699321,28cyclictest11743-21fschecks_count11:30:151
3233699321,27cyclictest4377-21ntpq10:30:241
3233699321,27cyclictest0-21swapper/107:35:141
3233699321,26cyclictest11544-21apt-get10:00:121
3233699320,28cyclictest13682-21irqcore10:55:181
3232499329,11cyclictest131rcu_preempt10:20:140
3232499325,18cyclictest12-21ksoftirqd/007:30:140
3235299314,18cyclictest35966irq/58-eth010:35:133
3235299313,24cyclictest35966irq/58-eth011:05:133
3234499318,7cyclictest18029-21diskmemload10:55:442
3234499312,11cyclictest22188-21ssh12:30:132
3234499311,8cyclictest8480-21ssh10:33:582
3234499311,8cyclictest28033-21kthreadcore09:50:202
3234499311,7cyclictest6277-21ps09:55:282
3234499311,5cyclictest14516-21/usr/sbin/munin07:25:262
32344993113,11cyclictest27-21ksoftirqd/211:05:132
3234499311,11cyclictest0-21swapper/211:54:162
3233699312,26cyclictest22178-21apt-get12:30:121
3233699311,28cyclictest11268-21unixbench_multi08:25:271
3235299304,13cyclictest35966irq/58-eth010:55:143
3234499302,7cyclictest18029-21diskmemload10:36:482
3234499301,9cyclictest2959-21grep09:35:302
3234499301,6cyclictest22482-21processes12:10:302
3234499301,11cyclictest3546-21cron08:55:012
3234499301,11cyclictest3546-21cron08:55:002
3233699301,26cyclictest27359-21cut08:45:201
3233699301,26cyclictest22635-21ssh12:10:301
3233699301,26cyclictest12405-21tune2fs08:30:161
3233699301,25cyclictest19305-21find07:30:121
3233699301,25cyclictest18823-21tail09:10:181
3233699300,27cyclictest25845-21ssh11:20:031
3232499302,13cyclictest0-21swapper/012:25:140
3234499296,3cyclictest18029-21diskmemload11:25:042
3234499291,7cyclictest1820966ssh12:09:532
3234499291,6cyclictest18029-21diskmemload09:26:132
3234499291,16cyclictest930-21snmpd10:05:132
3234499291,11cyclictest14847-21latency_hist11:50:002
3234499290,12cyclictest20621-21/usr/sbin/munin08:05:242
3232499295,16cyclictest12-21ksoftirqd/008:45:120
3235299286,18cyclictest33-21ksoftirqd/309:20:113
3235299286,18cyclictest33-21ksoftirqd/309:20:103
3235299281,16cyclictest35966irq/58-eth010:20:143
3234499285,3cyclictest20483-21sh11:35:052
3233699281,2cyclictest0-21swapper/109:00:141
3232499283,20cyclictest27635-21ssh12:15:140
3235299276,15cyclictest35966irq/58-eth009:55:123
3234499271,9cyclictest4022-21ssh09:20:112
3234499271,9cyclictest4022-21ssh09:20:112
3232499275,14cyclictest151rcuc/011:45:130
32324992710,7cyclictest131rcu_preempt10:30:120
3235299263,11cyclictest35966irq/58-eth009:35:133
3232499266,12cyclictest12-21ksoftirqd/010:40:150
3232499263,18cyclictest11564-21ssh10:00:140
32324992610,6cyclictest12-21ksoftirqd/009:00:130
3235299252,13cyclictest0-21swapper/312:35:113
3233699251,21cyclictest32320-21cyclictest07:10:021
32324992513,9cyclictest19839-21ssh09:45:220
3235299244,13cyclictest35966irq/58-eth010:40:153
3234499247,8cyclictest14454-21cat07:25:022
3232499242,14cyclictest22202-21ssh12:30:140
3232499241,19cyclictest31205-21ssh09:35:140
3232499241,18cyclictest2582-21ssh11:25:140
3235299233,13cyclictest35966irq/58-eth009:50:133
32352992312,5cyclictest0-21swapper/310:50:283
32352992310,9cyclictest35966irq/58-eth011:10:133
3232499237,13cyclictest151rcuc/009:20:110
3232499237,13cyclictest151rcuc/009:20:100
3232499232,16cyclictest12-21ksoftirqd/011:00:140
3232499232,14cyclictest0-21swapper/011:10:140
32324992313,5cyclictest0-21swapper/010:55:130
3235299229,10cyclictest0-21swapper/311:34:173
3235299226,5cyclictest0-21swapper/308:20:123
3235299223,13cyclictest0-21swapper/310:10:143
3235299223,13cyclictest0-21swapper/310:10:143
3235299222,12cyclictest35966irq/58-eth011:55:143
32352992212,3cyclictest0-21swapper/309:15:123
3232499226,9cyclictest12-21ksoftirqd/008:20:130
3232499221,2cyclictest0-21swapper/011:50:130
3235299213,12cyclictest0-21swapper/312:30:133
3235299212,13cyclictest0-21swapper/312:05:143
3235299211,15cyclictest35966irq/58-eth008:15:113
32352992110,7cyclictest0-21swapper/310:00:143
3234499218,5cyclictest1-21systemd08:40:132
3234499211,17cyclictest930-21snmpd07:15:122
3234499210,17cyclictest17574-21cut11:15:172
3232499212,14cyclictest844-21munin-node08:00:130
3232499211,9cyclictest738-21ntpq10:10:250
3232499211,9cyclictest738-21ntpq10:10:240
32324992110,7cyclictest5204-21ssh10:50:200
32324992110,3cyclictest131rcu_preempt12:00:010
3235299209,9cyclictest0-21swapper/309:40:283
32352992010,8cyclictest0-21swapper/310:49:303
32352992010,7cyclictest35966irq/58-eth010:15:113
3232499202,13cyclictest20641-21df_inode10:05:140
3232499201,17cyclictest12795-21latency_hist12:25:000
3235299192,11cyclictest0-21swapper/307:15:123
3235299191,12cyclictest0-21swapper/310:25:123
3235299191,12cyclictest0-21swapper/308:50:163
3235299191,12cyclictest0-21swapper/308:50:153
3235299191,12cyclictest0-21swapper/308:05:213
32352991910,6cyclictest35966irq/58-eth012:20:123
32352991910,6cyclictest35966irq/58-eth012:15:133
3232499196,9cyclictest20865-21wc08:05:160
3232499194,3cyclictest131rcu_preempt09:30:150
3232499193,11cyclictest0-21swapper/007:15:120
3232499192,14cyclictest29239-21grep07:40:180
3232499192,13cyclictest31224-21memory10:45:230
3232499191,9cyclictest15189-21kernelversion07:25:180
3232499191,13cyclictest7557-21apt-get11:10:020
3232499191,13cyclictest28770-21apt07:40:010
3232499190,16cyclictest9806-21cpuspeed07:20:150
3235299185,10cyclictest33-21ksoftirqd/309:00:283
3235299183,11cyclictest35966irq/58-eth012:25:153
3235299182,11cyclictest0-21swapper/310:30:133
3235299182,11cyclictest0-21swapper/309:25:143
3235299182,10cyclictest0-21swapper/308:00:143
3235299181,14cyclictest0-21swapper/311:00:213
3235299181,13cyclictest0-21swapper/308:55:283
3235299181,13cyclictest0-21swapper/307:30:013
3235299181,10cyclictest0-21swapper/307:55:023
32352991810,6cyclictest35966irq/58-eth007:30:143
32352991810,5cyclictest0-21swapper/309:30:173
3232499184,9cyclictest32411-21ls07:10:020
3232499181,12cyclictest26149-21ls09:50:170
3235299179,6cyclictest0-21swapper/311:54:143
3235299177,7cyclictest35966irq/58-eth011:20:133
3235299172,10cyclictest0-21swapper/308:35:143
3235299172,10cyclictest0-21swapper/307:55:143
3235299172,10cyclictest0-21swapper/307:55:143
3235299172,10cyclictest0-21swapper/307:35:043
32352991711,4cyclictest0-21swapper/312:03:053
3235299171,12cyclictest0-21swapper/308:25:273
3235299171,12cyclictest0-21swapper/307:45:223
32352991710,5cyclictest0-21swapper/309:10:223
3232499178,7cyclictest1568-21cut07:45:160
3232499176,8cyclictest25698-21unixbench_singl08:40:290
3232499172,12cyclictest18864-21kthreadcore08:35:210
3232499171,14cyclictest14683-21kthreadcore08:30:210
3232499171,14cyclictest12553-21kthreadcore07:55:200
3232499171,14cyclictest12553-21kthreadcore07:55:190
3232499171,12cyclictest1864566ssh09:10:170
3232499171,12cyclictest16866-21ntp_192.168.11409:05:250
3232499171,11cyclictest7793-21find09:40:110
3235299162,10cyclictest0-21swapper/308:45:143
3235299162,10cyclictest0-21swapper/308:10:143
3235299161,10cyclictest0-21swapper/307:40:153
3232499168,6cyclictest8359-21users08:55:300
3232499165,8cyclictest583-21sed07:10:180
3232499165,8cyclictest583-21sed07:10:180
3232499164,9cyclictest1046-21kthreadcore08:50:210
3232499164,9cyclictest1046-21kthreadcore08:50:200
3232499162,11cyclictest9173-21kthreadcore08:25:190
3232499161,3cyclictest131rcu_preempt08:10:120
3232499161,2cyclictest131rcu_preempt08:15:120
32352991510,3cyclictest0-21swapper/309:05:253
3235299142,9cyclictest0-21swapper/308:30:283
3235299141,9cyclictest0-21swapper/308:40:143
3234499142,10cyclictest32399-21cut07:10:012
3235299123,7cyclictest0-21swapper/307:10:283
3235299123,7cyclictest0-21swapper/307:10:273
3235299111,7cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional