You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-04 - 14:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sat Apr 04, 2026 12:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
15922996031,19cyclictest28366irq/58-eth010:00:143
15922995435,11cyclictest28366irq/58-eth009:40:133
1592299443,27cyclictest28366irq/58-eth011:25:133
1592299434,24cyclictest28366irq/58-eth011:20:113
1592299414,22cyclictest28366irq/58-eth012:05:133
15922994115,19cyclictest0-21swapper/311:09:163
1589799411,31cyclictest2377-21rm09:10:130
1589799411,31cyclictest2377-21rm09:10:130
1589799411,28cyclictest1967-21diskmemload11:10:140
15922994028,7cyclictest28366irq/58-eth010:35:133
1589799401,29cyclictest10893-21ssh10:45:130
1589799401,29cyclictest10893-21ssh10:45:120
1592299386,16cyclictest28366irq/58-eth010:20:143
1592299383,20cyclictest28366irq/58-eth008:40:133
1589799381,3cyclictest131rcu_preempt12:15:140
1589799381,27cyclictest23986-21ssh09:40:120
1592299377,14cyclictest28366irq/58-eth012:10:123
1592299372,24cyclictest28366irq/58-eth011:00:133
15922993420,5cyclictest0-21swapper/309:15:143
1589799331,25cyclictest3474-21irqcore11:35:170
15922993216,11cyclictest0-21swapper/312:20:133
1589799320,25cyclictest26520-21ssh11:30:120
1589799320,25cyclictest26520-21ssh11:30:120
1592299315,16cyclictest28366irq/58-eth010:05:143
1589799312,21cyclictest5566-21/usr/sbin/munin11:00:170
1592299306,13cyclictest0-21swapper/310:30:133
15914993020,7cyclictest261rcuc/210:20:122
1591499301,10cyclictest944866ssh11:55:222
1589799302,21cyclictest12323-21ssh11:40:160
1592299296,13cyclictest28366irq/58-eth008:05:123
1589799293,17cyclictest12-21ksoftirqd/010:55:140
1592299283,16cyclictest0-21swapper/311:40:163
15922992812,12cyclictest28366irq/58-eth011:45:123
1591499282,17cyclictest3055-21ssh11:35:112
1590699283,3cyclictest131rcu_preempt08:35:121
1589799283,20cyclictest0-21swapper/012:09:170
1592299274,12cyclictest28366irq/58-eth011:35:133
1592299273,15cyclictest0-21swapper/312:30:173
1592299271,16cyclictest28366irq/58-eth007:55:133
1591499272,21cyclictest12897-21grep08:10:282
15922992618,3cyclictest321rcuc/310:59:363
1591499267,3cyclictest881-21lldpd11:14:242
1591499263,15cyclictest2919-21kthreadcore09:45:212
1591499262,17cyclictest25679-21kthreadcore09:40:212
1591499262,16cyclictest16312-21latency_hist10:30:022
1591499261,7cyclictest112866ssh12:10:162
15914992614,8cyclictest0-21swapper/209:20:132
1591499261,21cyclictest10373-21latency_hist08:45:022
15897992612,10cyclictest1967-21diskmemload11:50:110
15897992610,10cyclictest12-21ksoftirqd/010:50:110
1592299257,8cyclictest0-21swapper/309:45:133
15922992517,6cyclictest321rcuc/309:34:163
1591499254,13cyclictest29937-21kthreadcore10:00:202
1591499252,20cyclictest30819-21latency_hist10:20:022
1591499252,15cyclictest4069-21apt-key08:05:012
1591499252,15cyclictest17891-21kthreadcore07:10:192
15914992512,10cyclictest31687-21ssh11:30:502
15914992512,10cyclictest31687-21ssh11:30:492
1591499251,19cyclictest11968-21apt-get11:40:132
1590699251,10cyclictest131rcu_preempt09:50:151
1589799251,19cyclictest738-21ssh12:10:130
1589799251,19cyclictest22193-21ssh10:15:140
1592299243,9cyclictest0-21swapper/309:20:123
1591499242,20cyclictest17214-21kthreadcore12:00:202
1591499242,14cyclictest25676-21/usr/sbin/munin10:35:162
15914992414,8cyclictest0-21swapper/212:35:252
15914992414,8cyclictest0-21swapper/212:05:132
15914992412,9cyclictest22381-21ssh11:09:342
15914992412,9cyclictest17932-21ssh12:19:412
15914992412,5cyclictest19801-21ssh09:19:532
1591499241,15cyclictest1205-21unixbench_multi09:05:272
1591499241,14cyclictest2764766ssh12:25:152
1591499241,11cyclictest1248966ssh11:03:232
1590699241,8cyclictest22228-21ntp_192.168.11409:55:251
1590699241,20cyclictest4065-21ssh12:30:111
1590699240,21cyclictest0-21swapper/107:55:121
1589799242,17cyclictest4404-21if_eth012:30:170
1589799242,15cyclictest27665-21diskstats12:25:170
1589799241,8cyclictest2702-21basename07:30:010
15897992412,6cyclictest11362-21ssh09:15:140
1589799241,18cyclictest17603-21ssh11:25:140
1589799241,18cyclictest14561-21cat11:05:180
1592299233,15cyclictest0-21swapper/311:50:133
1592299233,13cyclictest0-21swapper/312:25:143
15922992315,5cyclictest321rcuc/310:27:123
15922992314,3cyclictest321rcuc/311:10:533
1591499238,12cyclictest0-21swapper/210:55:142
1591499232,18cyclictest31754-21cut09:25:232
1591499232,18cyclictest14559-21munin-plugin-st09:35:012
1591499232,13cyclictest23891-21irqcore08:25:162
1591499231,19cyclictest15936-21latency_hist07:10:012
1591499231,19cyclictest15936-21latency_hist07:10:012
15914992310,6cyclictest28112-21ssh11:48:132
1590699238,3cyclictest131rcu_preempt09:35:141
1590699237,3cyclictest131rcu_preempt10:15:291
1590699233,6cyclictest131rcu_preempt09:15:141
1590699233,17cyclictest18601-21ssh12:20:111
15906992313,7cyclictest23569-21apt-get08:25:131
1589799231,3cyclictest131rcu_preempt09:35:130
1589799231,14cyclictest27634-21ssh10:00:130
1589799230,11cyclictest1952-21ssh10:40:130
1592299222,12cyclictest0-21swapper/309:10:143
1592299222,12cyclictest0-21swapper/309:10:133
15922992213,6cyclictest321rcuc/309:35:203
15922992211,8cyclictest0-21swapper/310:40:143
1591499228,5cyclictest11212-21cat10:45:172
1591499228,5cyclictest11212-21cat10:45:172
1591499227,12cyclictest31661-21cron08:00:022
1591499223,15cyclictest5613-21latency_hist08:40:012
1591499223,12cyclictest14788-21kthreadcore07:40:202
1591499223,12cyclictest12349-21idleruntime-cro07:40:012
1591499222,17cyclictest26873-21idleruntime-cro07:55:002
1591499222,17cyclictest14475-21kthreadcore10:10:182
1591499222,16cyclictest4146-21/usr/sbin/munin10:05:252
1591499222,13cyclictest22521-21wc08:20:292
1591499222,12cyclictest3839-21kthreadcore07:30:182
1591499222,11cyclictest18616-21ssh12:20:132
15914992211,9cyclictest24595-21ssh11:28:072
1591499221,18cyclictest23529-21missed_timers08:55:222
1591499221,17cyclictest26363-21cut09:00:192
1591499221,17cyclictest14411-21irqcore08:15:182
15914992211,3cyclictest20610-21tail10:30:292
1591499221,11cyclictest15105-21sort09:35:172
15914992210,4cyclictest21860-21kthreadcore10:50:202
1591499220,13cyclictest1642866ssh11:24:072
1590699228,3cyclictest131rcu_preempt12:00:191
1590699224,4cyclictest21-21ksoftirqd/109:25:251
1590699223,16cyclictest23953-21/usr/sbin/munin09:40:231
1590699222,17cyclictest8618-21ssh11:20:111
15906992217,3cyclictest0-21swapper/109:00:131
15906992214,5cyclictest0-21swapper/111:35:111
1590699221,18cyclictest30495-21grep11:30:261
1590699221,18cyclictest30495-21grep11:30:261
1589799221,18cyclictest7549-21ssh10:25:130
15922992116,3cyclictest321rcuc/312:15:453
15922992116,3cyclictest321rcuc/311:30:423
15922992116,3cyclictest321rcuc/311:30:413
15922992115,3cyclictest321rcuc/312:02:213
1592299211,12cyclictest0-21swapper/308:35:123
1591499213,11cyclictest5347-21kthreadcore09:10:222
1591499213,11cyclictest5347-21kthreadcore09:10:212
1591499213,11cyclictest29382-21kthreadcore08:30:192
1591499213,11cyclictest2663-21irqrtprio10:40:182
1591499212,16cyclictest19083-21irqcore09:55:192
1591499212,11cyclictest25584-21latency_hist07:20:022
1591499211,8cyclictest3010966ssh11:50:152
1591499211,17cyclictest30118-21/usr/sbin/munin07:20:302
1591499211,17cyclictest1193-21kthreadcore07:25:222
15914992110,9cyclictest17013-21ssh09:53:412
1590699219,3cyclictest21-21ksoftirqd/110:09:011
1590699215,3cyclictest131rcu_preempt08:10:141
1590699212,17cyclictest2611-21awk10:20:291
1590699212,16cyclictest28120-21cat12:05:281
1590699212,16cyclictest21524-21cron12:40:021
15906992114,4cyclictest390-21unixbench_singl10:55:291
15906992112,6cyclictest25463-21cron07:20:021
15906992110,8cyclictest20603-21cron11:45:021
1589799213,15cyclictest29353-21ssh09:25:160
1589799213,11cyclictest0-21swapper/008:05:140
1589799211,6cyclictest0-21swapper/008:35:120
15897992114,4cyclictest19397-21unixbench_singl12:00:280
1589799211,13cyclictest25706-21find07:20:130
1589799211,11cyclictest18077-21apt-get09:55:020
1592299203,12cyclictest0-21swapper/310:50:143
15922992015,3cyclictest321rcuc/307:25:183
15922992014,3cyclictest321rcuc/308:15:023
1591499203,14cyclictest17552-21fschecks_time07:45:172
1591499202,11cyclictest20066-21latency_hist08:55:022
1591499202,11cyclictest13108-21kthreadcore08:45:212
1591499201,5cyclictest402466ssh12:30:112
1591499201,11cyclictest32336-21uniq11:15:172
1591499201,10cyclictest7813-21awk08:05:252
1590699208,2cyclictest131rcu_preempt10:30:151
1590699205,3cyclictest131rcu_preempt11:50:211
1590699204,2cyclictest0-21swapper/111:55:131
1590699204,14cyclictest10479-21kthreadcore12:15:211
1590699204,13cyclictest25441-21acpi10:35:121
1590699204,13cyclictest16065-21cron10:30:011
1590699203,15cyclictest4875-21grep08:05:181
1590699203,15cyclictest25104-21kthreadcore11:10:211
1590699203,14cyclictest31289-21kthreadcore09:05:181
1590699203,14cyclictest14419-21irqcore08:15:181
1590699202,15cyclictest16313-21ssh10:10:201
1590699201,7cyclictest31108-21memory10:00:231
15906992015,3cyclictest0-21swapper/111:00:141
1590699201,16cyclictest6422-21irqcore09:30:171
1590699201,16cyclictest6314-21find09:10:291
1590699201,16cyclictest6314-21find09:10:291
1590699201,16cyclictest5589-21awk10:40:241
1590699201,16cyclictest1623-21kthreadcore12:10:191
15897992014,4cyclictest0-21swapper/008:00:170
15897992013,4cyclictest0-21swapper/007:30:170
15897992012,6cyclictest25842-21sshd09:21:170
15897992012,5cyclictest0-21swapper/007:55:060
15897992011,6cyclictest0-21swapper/008:25:230
15897992010,7cyclictest9438-21cut11:20:180
1589799200,8cyclictest1967-21diskmemload09:55:300
1592299193,10cyclictest0-21swapper/309:00:123
1592299192,10cyclictest0-21swapper/308:30:123
15922991914,3cyclictest321rcuc/309:50:153
15922991913,4cyclictest0-21swapper/311:18:133
15922991913,4cyclictest0-21swapper/310:12:043
15922991913,4cyclictest0-21swapper/309:25:043
15922991912,3cyclictest0-21swapper/310:45:353
15922991912,3cyclictest0-21swapper/310:45:353
15922991911,5cyclictest0-21swapper/309:55:173
1592299191,13cyclictest0-21swapper/310:15:283
1592299191,11cyclictest0-21swapper/307:10:233
15922991910,6cyclictest28366irq/58-eth011:55:133
1590699199,7cyclictest18744-21ntpq08:50:261
1590699194,12cyclictest32039-21fschecks_count08:00:161
1590699193,14cyclictest21960-21munin-run07:50:001
1590699193,14cyclictest1869-21timerwakeupswit07:25:271
1590699192,4cyclictest131rcu_preempt11:05:151
1590699192,15cyclictest32656-21irqcore11:15:181
1590699192,15cyclictest31316-21kthreadcore08:30:201
1590699192,15cyclictest24775-21grep07:15:291
1590699191,5cyclictest0-21swapper/109:45:061
1590699191,16cyclictest21546-21kthreadcore08:55:181
1590699191,16cyclictest16379-21grep07:40:281
1590699191,15cyclictest3130-21fschecks_time07:30:161
1590699191,15cyclictest31184-21sed12:25:271
1590699191,15cyclictest0-21swapper/107:35:141
1589799196,10cyclictest10798-21grep08:45:170
1589799193,11cyclictest0-21swapper/007:40:140
1589799191,7cyclictest457-21kthreadcore09:05:210
1589799191,7cyclictest16704-21tune2fs10:30:150
1589799191,6cyclictest0-21swapper/008:40:120
15897991914,3cyclictest0-21swapper/008:55:150
15897991913,4cyclictest0-21swapper/008:50:270
15897991912,5cyclictest6185-21if_err_eth009:30:150
1589799191,15cyclictest501-21rm09:45:130
1589799190,16cyclictest20976-21ssh11:45:130
1592299182,10cyclictest0-21swapper/308:55:043
1592299182,10cyclictest0-21swapper/307:50:153
15922991814,3cyclictest321rcuc/308:00:073
15922991812,4cyclictest321rcuc/307:35:133
1592299181,13cyclictest0-21swapper/307:40:283
1590699189,6cyclictest20223-21sshd10:50:181
1590699182,14cyclictest10471-21acpi08:45:121
1590699182,13cyclictest18990-21fschecks_time08:20:161
1590699181,15cyclictest16255-21fschecks_count07:10:161
15906991810,6cyclictest26559-21sshd11:46:181
1589799189,7cyclictest22357-21df07:50:150
1589799188,3cyclictest12-21ksoftirqd/012:35:290
1589799187,8cyclictest12-21ksoftirqd/012:20:240
1589799182,11cyclictest0-21swapper/008:15:130
1589799181,5cyclictest0-21swapper/007:45:130
1589799181,4cyclictest4571-21ssh10:05:190
15897991811,5cyclictest12301-21sshd07:37:490
1592299179,6cyclictest0-21swapper/312:35:593
1592299174,7cyclictest0-21swapper/307:45:133
1592299172,9cyclictest0-21swapper/307:30:143
15922991712,3cyclictest321rcuc/307:15:133
15922991712,3cyclictest0-21swapper/309:05:163
15922991711,4cyclictest0-21swapper/307:20:303
1590699179,6cyclictest18308-21uname11:25:201
1590699177,7cyclictest11300-21sed10:45:181
1590699177,7cyclictest11300-21sed10:45:171
1590699173,12cyclictest23832-21kthreadcore07:50:211
1590699172,12cyclictest7393-21kthreadcore08:40:211
15906991713,2cyclictest0-21swapper/109:20:111
1589799178,7cyclictest6507-21df_inode11:55:140
1589799176,7cyclictest20823-21latency_hist07:15:000
1589799172,4cyclictest0-21swapper/008:35:010
1589799172,12cyclictest31776-21irqstats10:20:190
1589799171,7cyclictest1967-21diskmemload10:38:180
15897991711,4cyclictest0-21swapper/009:00:210
1589799171,13cyclictest32110-21ssh11:15:120
1592299162,11cyclictest0-21swapper/308:55:013
1592299162,10cyclictest0-21swapper/308:45:143
1592299162,10cyclictest0-21swapper/308:25:133
1590699161,12cyclictest16053-21latency_hist07:10:021
1590699161,12cyclictest16053-21latency_hist07:10:011
1589799162,9cyclictest0-21swapper/008:20:150
1589799162,10cyclictest0-21swapper/007:15:150
1589799161,11cyclictest13217-21df_inode10:10:130
15922991510,3cyclictest0-21swapper/308:24:303
15922991510,3cyclictest0-21swapper/308:15:133
1589799151,11cyclictest9173-21apt-get08:10:120
1589799131,9cyclictest15851-21latency_hist07:10:010
1589799131,9cyclictest15851-21latency_hist07:10:000
1592299111,6cyclictest0-21swapper/307:10:023
1592299111,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional